Value for Money Literature Review September 2016
Value for Money Literature Review September 2016
Literature Review Brief
1.0 Introduction Following the brief provided for this Review, and a discussion of emerging themes with the working group, five key areas were identified for this piece of work: Conceptual Challenges: The report identifies key questions to think about when considering VfM:
Who is involved? Whose values are being measured? How are these going to be measured?
It outlines the link between VfM and accountability, and asks readers to consider who are they providing accountability for – funders, service users, or both?
Value for Money across the UK: The report looks at the different approaches across England, Scotland, and Wales
England – approach is based on unit cost, and economic performance Scotland – focus is on tenants and developing a crosssector approach to measuring VfM Wales – emerging focus on tenant engagement and capacity to align with other Welsh Government priorities
Value for Money approaches in different public services: International Development
Emphasis on ‘equity’ The measurement of long term outcomes on individuals’ lives following investment in services 1|Page
Health
Looking at quality of life for individuals in relation to cost of treatments What are they able to do and be?
International approaches to Value for Money: The report will look into international approaches to value for money, focussing specifically on:
The Netherlands – portfolio analysis, the systems approach and scale and value for money Australia – research undertaken to assess management costs and tenant outcomes in social housing
Social Value:
Definitions of social value Monetary and non-monetary approaches Models of social value
How to use this document:
The Review is designed to be easy to use and accessible to a wide range of audiences. There are ‘Key Points’ sections for each chapter – these are there to communicate the key messages of the chapter, without the detail. Having read the ‘Key Points’, you can read the fuller detail if you are particularly interested in this section There are diagrams throughout to show visually some of the issues we discuss in the text
2|Page
2.0 Conceptual Introduction
The 1970s saw a large shift in the approach to management practices within public sector organisations in the UK. At the time, these organisations were seen as lacking accountability, being wasteful with resources, and therefore failing to provide value for money. To combat this, private sector management practices were introduced to give a more rigorous approach to the use of resources and delivery of outputs (Alwardat, Benamraoui and Rieple 2015). The concept of VfM in public expenditure has been further focused on since the HM Treasury’s 2003 publication of ‘Green Book’ which provided a framework by which to evaluate and appraise all government policy and intervention. Within the context of an ‘austerity agenda’ in the UK following the 2008 financial crash, questions around the efficient use of public funding and providing accountability for this have been prevalent within government and the public sector. The discussion of Value for Money can be seen as a response to these issues. The business of the public sector is broad, and discussing the concept of Value for Money across the whole spectrum of public services can give us a range of different models, approaches, and interpretation of the concept itself. It is important to be clear, therefore, on the understanding of the concept we are engaging with before analysing different types of models and impacts. The National Audit Office defines VfM as ‘the optimal use of resources to achieve the intended outcomes’ and there is a general consensus that the measurement relates to the level of resources committed to an action, and the output of this action. As Smith (2009, p. 6) outlines, in its abstract form the concept is straightforward “it represents the ratio of some measure of valued…system outputs to the associated expenditure”. This general definition can be seen to relate to the idea of ‘the Three Es’; economy, efficiency, and effectiveness.
3|Page
Economy
Delivering product
At lowest cost
Efficiency
Achieving the biggest impact
From resources used
Effectiveness
Making sure product is fully delivered
From the use of the resources
The first category relates to the level of resource put into the activity; this should be the minimum cost by which the activity can be completed. The second relates to achieving the maximum output from this minimum input. This entails getting the most out of the resources used. The final category works to ensure that the intended result is fully attained from the use of the resources. VfM focusses on the relationship, therefore, between the level of resource used to complete an activity, and the level and effectiveness of this output. The test for VfM is whether an appropriate level of resources has been used to achieve a particular result. When looking at public sector expenditure we can ask whether the level of funding used to deliver a particular project reflects the impact that this project has had. A third ‘E’ has been added to this framework; equity. This addition emphasises activities only have value if they are done fairly. For example, healthcare provision must fairly assess individuals’ needs with patients having equitable access to services. The use of this addition is prevalent in the Development Sector, which will be discussed in greater detail in a following chapter. The broadest description of why VfM is important is ‘accountability’. Funders (in the form of government, organization, or taxpayer) need to understand where their money is going and the impact that this is having. Within this idea of ‘upward accountability’ there is a distinction between accountability to funders who have been part of deciding what the activity that should be undertaken should be, and funders who have no direct link with deciding this such as general taxpayers. ‘Downward accountability’ ensures that beneficiaries of the activity understand what decisions have been made and why. Particularly in the development sector there has been a risk of an increased push towards upward accountability rather than downward; ensuring that justification for spending is made to donors but not to the individuals and communities where aid is being provided. 4|Page
Upward accountability Providing justification to funders
Downward accountability Providing justification to people recieving services
As already highlighted, there is a large range of approaches taken within the different areas of the public sector. The move from the theoretical model of VfM to one that can be applied in practice, therefore, can be seen as challenging. It is important to understand a number of key areas of discussion within the idea of VfM to both understand this variation in approach, and be able to create new models. The core ideas here are:
Who? • Who are the people involved in the service? • How do they differ?
Whose values? • When we measure VfM whose values are we measuring? • How do they differ?
How? • We've decided on the values we are going to look at... • ...how do we measure them?
Who? It is vital to understand who the actors are within the service or project that is being measured for VfM. Within a public sector activity this could include governmental departments who provide funding, policy-makers who decide how it is spent, delivery agents who facilitate the activity, 5|Page
and beneficiaries for which the activity is meant to make an impact. These beneficiaries could either be individuals or communities more broadly. These groups might all have different interests which will need to be taken into account when a development for VfM is being created. It is important to relate this to the discussion of accountability. When creating an approach to measuring VfM are you looking to provide upward accountability, downward accountability, or a combination of the two? Whose values? It is important to consider who these actors are, therefore, when understanding what values you are measuring:
Are they defined by the policy-makers, the funders, or the beneficiaries? And who should be defining that value which is being measured?
Again, it is important to be able to answer these questions to be able to reflect on the type of accountability – downward or upward or both – that the VfM measurement is providing. How? Both quantitative and qualitative values are measured through a range of VfM approaches, and it is important to understand what type of value is being measured: is it a ‘hard’ or ‘soft’ value? Hard values can be broadly understood as something you can count; a level of funding made available, or the number of jobs created from a particular project. Economic goods, or the equivalence of an activity to an economic saving has been the general approach to VfM in a number of areas. One approach can be that “monetarizing things give them value” (Emmi et. al. 2011, p. 15) whereas not monetarizing impacts or activities can be seen to be synonymous with not valuing them. By measuring VfM through quantitative methods and ‘hard data’ predictions and comparisons can be drawn. It is also easier to communicate impact ‘upward’ to decision-makers and taxpayers. Glendinnig (1988, p.45) argues, however, that “non-economic values lie close to the heard of satisfactory living and it is these with which VfM is concerned”. These non-economic values can be seen as ‘soft’ values such as individuals gaining confidence to apply for jobs, or take on volunteering roles. This can be seen to chime with previous discussions around the actors involved in the activity which is being measured, and the type of accountability which is being provided. Concerns have been raised that “programmes that are most precisely and easily measured are the least transformational” (Emmi et. al. 2011, p. 16). This discussion around the type of values being measured will 6|Page
be returned to in further sections on VfM case studies, and social value. From looking at one method of evaluating VfM we can see the different range of approaches to how and what we measure. Trosa and Williams (1996) outline three different approaches taken to benchmarking public sector spending; standards, results, and process. The first type sets a standard of performance which organisations should expect to achieve. The second compares the performance of organisations which provide a similar service. This is done in England and Wales by the police forces whose performance is compared with a similar type of force. The final type looks at performance within a group of organisations to understand the process by which an output was achieved, and how good practice can be shared. All three measurements are therefore different, although they are part of the same process. The outcomes from these measurements will therefore be different, and show different things. In some cases one type of measurement will be appropriate where another isn’t. The decision around what values are to be measured in a VfM calculation, then, can be seen to be complex. There needs to be an understanding of who the actors in the activity are; whose values are being measured; what type of values these are; and how they are being recorded and evaluated. The final area to consider is the role of the organization which is enforcing VfM measurement on organisations. As Glendinnig notes, there is a difference in approach to VfM between public and private sector bodies in terms of motivation. For private sector businesses the motivation is clear; if they do not provide VfM their customers will go elsewhere. This is different for public sector bodies, where ‘customers’ have limited choice due to the nature of the goods being provided and the way it is subsidized by government. VfM measurements in the public sector are often compulsory, therefore. However, there are different ways in which this can be done. Within the Higher Education sector, the funding body HEFCE sees its role as enabling universities to be able to provide VfM. The organisation asks universities to embed this form of evaluation within the structure of institutions as a proactive way of ensuring that the best service is delivered to students. Within the English housing sector, however, a very different approach is taken to VfM measurements with the Housing Regulator taking a more forceful approach. There can be seen to be two key approaches to implementing VfM measures:
Enabling – the approach to VfM allows organisations to deliver the best service they can for the resources they use
7|Page
Enforcing – the approach to VfM demands that organisations meet a certain standard in regards to the relationship between resources and outputs
KEY POINTS
A number of conceptual ideas have been outlined in this section which will be drawn on in further discussion of different approaches within housing, outside of housing, and outside of the UK. It is important to consider these ideas when thinking about the type of approach to VfM which should be taken in Wales.
1. Value for Money can be understood in a variety of different ways within the public sector 2. There are therefore a range of models available to measure it 3. VfM can be linked to the concept of accountability 4. It is important to consider who this accountability is for: funders (upward), service users (downward), or both? 5. There are key questions to consider when thinking about VfM: - Who is involved? - Whose values are being measured? - How are these being measured?
8|Page
Value for Money Literature Review September 2016
Introduction
2.0 VfM Across the U.K As already outlined, England has taken a very specific approach to VfM in its social housing sector. In this section, a review will be provided of the different approaches taken to VfM by the Regulators in England and Scotland, followed by some suggestions of how a ‘Welsh approach’ to the measurement could be formulated. The differences can be seen below:
The
Homes and Communities in England is facing uncertain times, with changes to regulatory functions in the Housing and Planning Act and shift in the status of Housing Associations from the ONS. All of these impact on the debate around VfM, particularly in regards to the nuance of who the actors are within measurement activity and how VfM is pursued by the Regulator. The approach taken in England is one which heavily emphasises compliance over enablement, and VfM has recently been more heavily embedded in the Regulator’s approach through an update to ‘Regulating the Standards’ in July 2016.
England Agency
The HCA has two fundamental objectives for regulation; an economic regulation objective and a consumer regulation objective. The first 9|Page
objective incorporates measures around VfM as well as ensuring that providers are financially viable. This also ensures that an unreasonable burden is not placed on public funds and that these are not misused. The second objective focuses on the quality and management of social housing stock, as well ensuring that tenants have choice and protection and the opportunity to participate in management and accountability. Within these two broader objectives sit more specific standards. Within the economic objective these are: Governance and Financial Viability, Value for Money, and Rent. Providers are to be measured against these standards so the HCA can “gain a strategic and evidenced understanding of both the short-term and longer-term risks to which providers are exposed, and to gain a comprehensive understanding of their approach to value for money” (HCA 2016a, p. 8). The Standards emphasises the importance of managing resources according to the ‘Three Es’ and in a way which takes into account the interests of and commitments to stakeholders. As well as demonstrating an understanding of the use of its resources and how this relates to returns on assets, the providers’ boards are required to demonstrate to stakeholders how they are meeting this standard. Compliance with this Standard is now a core part of the HCA’s In Depth Assessments. These are undertaken every three to four years for providers which own over 1000 properties, and the frequency is dependent on the risk profile of the organisation. It is important to note that in this approach the focus is on unit cost. There has been a renewed emphasis on this in the past few months through a number of publications, including a letter from the Regulator to all Housing Associations criticising the disparity in unit costs across the sector. As part of the IDAs the Regulator will now “seek assurance that providers understand unit costs derived from accounts data and, importantly, the reasons why they are higher or lower than other providers” (HCA 2016b, p. 2). Although there is the specification that VfM measurements must take into account stakeholders’ interests, and that there must be a demonstration to stakeholders of how the organization meets this standard, there is no specific mention of tenants. Only the second objective, on consumer regulation, includes duties regarding tenants. Within this there are 4 standards relating to possible or current tenants; Home, Tenancy, Neighbourhood and Community, Tenant Involvement and Empowerment. Instead of having regular IDAs where compliance with these standards are tested, the approach taken to the consumer regulation objective is reactive only. The HCA (2016a, p. 16) states “we only use our regulatory and enforcement powers where we judge that there has been a breach of a consumer standard which has or could cause serious detriment”. 10 | P a g e
Relating this back to the previous discussion on ‘upward’ and ‘downward’ accountability, it could be argued that the approach taken in England focusses on ‘upward’ accountability to funders but neglects ‘downward’ accountability to tenants. By leaving all mention of tenants to the reactive consumer-standard there is no duty for tenants to be incorporated into discussions of VfM. Furthermore, the focus of unit cost within the economic standards Scotland indicates a reliance on looking at quantitative measures and ‘hard data’ as the sum measurement of VfM. Scotland, on the other hand, is taking an alternative approach to measuring VfM. The Scottish Housing Regulator has made tenants the primary focus of its perspective of VfM and this has been supported by an influential report by HouseMark, CIH Scotland, and the Wheatley Group. This piece of research engaged with the sector in Scotland regarding the practical experience of implementing VfM and aimed to set out “an optimum approach that individual organisations can select from and adapt depending on their size, geography, and purpose” (2015, p. 1). The report outlines a series of recommendations including:
A broadening of consideration of tenants to include future tenants and communities at large Encouraging housing providers to collect additional data voluntarily, rather than adding more requirements to the Annual Return on the Charter The Regulator should offer the Scottish housing sector the opportunity to develop its own perspective on VfM either before or as an alternative to creating a more prescriptive approach
Emphasising the final recommendation, the research outlines that the sector has “got the VfM message” and is able to develop a “bottom up” (2015, p.3) approach which would avoid the need for an increased regulatory burden. The report outlines a number of principles which should underpin an approach to VfM in Scotland, and proposes that a suite of measures be created to inform a VfM scorecard created by the sector. HouseMark outline that there are three core aspects to VfM which should be considered in tandem; defining the concept, managing it, and demonstrating it. The report outlines that the current regulatory approach in Scotland incorporates these aspects. Housing providers must show that they meet the standards within the Scottish Housing Charter. This document contains a range of standards which describe the results that tenants and customers want social landlords to achieve, cover only activities relating to social landlords, and can be 11 | P a g e
assessed and reported on by the Regulator. Outcome 13 in the document the VfM Standard:
Social landlords manage all aspects of their businesses so that: tenants, owners and other customers receive services that provide continually improving value for the rent and other charges they pay
There is an expectation that landlords self-assess and report performance according to this standard which meets the ‘demonstrating VfM’ criterion. Finally, demonstrating VfM also informs a dialogue around rent setting and the rent and service charge outcomes within the Charter also form a part of the VfM approach in Scotland. The report outlines a number of checklists for each three of the aspects that have been outlined above. Regarding managing VfM, the research distinguishes between two questions: 1) Do you do the right things to maximize value for your stakeholders? 2) Do you do things right? The first refers to whether the approach taken by the housing provider matches tenant priorities and whether this perspective of VfM has been balanced with other stakeholders’ views. The second focuses on whether the organization has an understanding of the cost and performance of services and assets, as well as general performance and financial management. The emphasis here is clearly on tenant engagement. We can see how this approach differs from the English approach where the Regulator takes a reactive rather than proactive approach to tenant engagement. The Scottish model with the emphasis placed on the tenant by the Regulator, as well as the tenant-focused suggestions within the HouseMark research, can be seen to provide a balance of both ‘upward’ and ‘downward’ accountability. The proposal of the creation of a scorecard fits into the aspect of ‘demonstrating VfM’. Here the research outlines that organizations need to create a suite of measures which help them to map the values (outputs) that they create through activities, help to understand the concept of ‘money’ or inputs into the activity, and third give an indication of the effectiveness of translating inputs to outputs. Within this framework, the research sets out a number of sets of measures that could be considered. These are:
Business processes E.g. rent/service charge collected People E.g. sickness absence Corporate Health E.g. growth turnover/operating margin 12 | P a g e
The report emphasises the importance of communicating, as well as measuring, VfM. It states “landlords should…reflect on the extent to which the VfM information they provide for tenants is accessible and transparent so that meaningful involvement in shaping services and scrutiny can take place” (2015, p. 25). The research notes that a number of regulatory downgrades that have occurred for English associations relate to the lack of communication of VfM measures and impact. Lessons should be learnt from this for the Scottish model, with an emphasis placed on clarity around the process and opportunity for all tenants to engage with, and challenge, the information. The report concludes by re-emphasising the need to define, manage, and demonstrate VfM. It outlines that as private companies seek to maximize values for shareholders, “social business should be looking to provide as much value as possible for stakeholders. At the heart of this mission are tenants and other service users” (2015, p. 27). It proposes that developing a ‘bottom up’ scorecard would enable this to happen, with the input of both the tenants and the Regulator. The process of measuring VfM allows businesses to demonstrate that they know why they exist, they know who they serve, what it needs to do to be successful, and have the ability to measure and demonstrate its success. The Scottish Social Housing Charter is currently under consultation, with the renewed st Wales Charter to be in place by 1 April 2017. The revisions will show whether the tenant focus from both the sector and the Regulator will remain in place for VfM measurement in the sector. Community Housing Cymru and HouseMark have recently published a report on developing a new approach to VfM in Wales, responding to the Regulatory Board for Wales’ focus on this for 2016. This report builds on their Scottish approach but in a Welsh context, highlighting how this type of VfM model could be used to contribute to wider Welsh Government priorities such as the Wellbeing of Future Generation’s (Wales) Act. Again, the report identifies that communicating VfM is as important as measuring it and emphasises the co-regulation approach taken in Wales. In the report, HouseMark outline the same key approaches as in the Scottish model but draws on the specific legislative and operating environment in Wales. It emphasises that within the approach taken to co-regulation in Wales, Housing Associations have an opportunity to take the lead in developing an approach to VfM. They suggest that the scorecard model could be used to do this, and suggest a number of key areas that could contribute to this:
13 | P a g e
1)
Effectiveness – measures such as: Units developed as a percentage of current stock Percentage satisfaction with new how Average SAP rating
2) Economy – measures such as: Cost per property of housing management, repairs and voids, overheads Overheads as a percentage of revenue spend Debt per unit 3)
Efficiency – measures such as: Percentage of arrears Percentage of void loss Average re-let time
4) Additional Social Value Data The report outlines the importance of utilising data that Housing Associations have already collated and reported on in order to provide measurement of VfM but minimise additional work. As previously noted, the report emphasises the role that this approach could play in demonstrating the ‘value-added’ aspect of the housing sector by showing how the work of the sector impacts on the Welsh Government’s priority around ‘The Wales We Want’. Using data that has already been reported on, the sector can show that not only has it met its own objectives but that these map on to National Indicators or goals. These include areas such as health, the environment, education, and poverty/deprivation. This proposed approach for Wales therefore emphasises the important role for tenants to play within the measurement of VfM (building on the Scottish model). It outlines that the housing sector should be a key driver of change in this area, and that financial metrics should be supported through ‘softer’ measures that might be included in an evaluation of social value.
14 | P a g e
England Who?
Upward accountability Yes
Downward accountability
Economic Objective focusses on demonstrating VfM in terms of financial inputs and outputs Tenants are only included in Consumer Regulation Objective which is reactive only
Whose values?
No tenant involvement in creation of VfM tool Focus on demonstrating VfM to funders rather than recipients
How?
Scotland –
Quantitative methods – financial indicators
Upward accountability Yes – duty created by
Downward accountability Yes – emphasis on 15 | P a g e
HouseMark report
Regulator
Who?
Housing Associations Tenants involved in and Regulator involved creating approach in creating approach
Whose values?
Broad discussion of organisations doing the ‘right thing’ to maximise value for stakeholders Scorecard of range of measurements including business processes
Tenants referred to throughout approach as vital to understanding VfM
Upward accountability Yes – duty created by Regulator
Downward accountability Yes – emphasis on including tenants
Who?
Emphasis on coregulation approach for Housing Associations, Housing Sector more broadly, and Regulatory team
Tenants involved in creating approach
Whose values?
Broad discussion of organisations doing the ‘right thing’ to maximise value for stakeholders
Tenants referred to throughout approach as vital to understanding VfM
How?
Wales – HouseMark Report
including tenants
Scorecard of range of measurements – with tenants involved in the creation of the scorecard
Alignment of meeting objectives with broader National Indicators How?
Scorecard of range of measurements – discussing effectiveness,
Scorecard of range of measurements – with the additional of social value measurement 16 | P a g e
KEY POINTS
economy, and efficiency
1. Approaches to VfM in social housing differ across GB 2. England and Scotland have clear, and very different, approaches - England - cost focus - Scotland - tenant focus 3. There is space for Wales to develop its own approach to VfM in line with other policy priorities
17 | P a g e
Value for Money Literature Review September 2016
3.0 Other Sectors
Introduction
Approach from
As previously outlined, the concept of VfM analysis became widespread within the public sector with the introduction of the New Public Management approach to government spending. In this section, a review will be provided of analysis frameworks and tools used within the development and health sectors respectively. This will draw on the concept of accountability as highlighted in the first section of the report, and build on the idea of ‘downward accountability’ as seen with the approaches outlined by HouseMark regarding tenant engagement.
Department for VfM analysis is an important tool within the development sector, with International much discussion on the use of aid spending and how the impact of the Development use of that can be measured ongoing. In this section, the approach taken by DFID will be outlined with an emphasis placed on its addition of ‘equity’ to The Three E’s. This will be done through the review of a report into the Value for Money within the Department’s WASH programmes. The inclusion of ‘equity’ as additional to the ‘Three Es’ demands that both sustained actual outcomes and impacts are measured as part of the Department’s analysis of its projects. DFID describe VfM as “maximising the impact of each pound spent to improve people’s lives” (Tremolet et. al. 2015, p. 1) and note how this builds on the National Audit Office’s definition of looking at actual outcomes for people. The report highlights that VfM for DFID is not necessarily about cost saving or lowering unit costs, but “is about maximising actual outcomes and impacts” (Tremolet et. al. 2015, p. 1). It is apparent, therefore, that the measurement included in this analysis includes a focus on the impact of the spending on individuals. This concept of equity makes the link between purely financial analysis of VfM and the impact that spending can have on the lives of individuals in terms of what they are able to be and do. The Department states that VfM analysis should not be just about a quantitative measurement but engage with stakeholders of the programme also. This can be seen to link to its emphasis on equity. 18 | P a g e
The results chain used for measuring VfM in water, hygiene, and schools programmes (WASH) looks like this:
There are a number of interconnected stages within this chain:
Costs – the money spent on an intervention Inputs – the capital or labour used in delivering this intervention Process – what happened through the intervention Outputs and Assumed Outcomes – these relate to either ‘hard outcomes’ that have been achieved, or the estimated number of individual affected by the intervention Sustained actual outcomes – “the actual change in poor people’s lives over time” (p. 6). Impacts – longer term changes to individuals and communities such as an increase in school attendance
The first four of these indicators can be seen to cover The Three E’s. Additional to these three key dimensions of VfM, DFID add costeffectiveness and cost-efficiency. The first relates to the relationship between the programme costs and the impact on beneficiaries over time. The second relates to the costs per output or assumed beneficiary. This takes into account the sources of funding for the intervention which led to the output or assumed outcome. It is the addition of equity, however, that calls for a measurement of the sustainable change that has been made in the lives of individuals or communities. The report into the WASH programmes states “sustainability and equity are considered as an additional layer of analysis that cuts across the main VfM dimensions (Tremolet et al. 19 | P a g e
2015, p. 6). This relates back to the discussions around actors, metric, and accountability within the first section of the report; when looking at VfM it is crucial to understand who the actors are in the process, what you are measuring, and why and who you are providing accountability to. In this case, DFID have already emphasised the importance of looking at the tangible impacts on individuals’ lives through interventions, and expanded the metric beyond one of purely quantitative data. Both ‘upward’ and ‘downward’ accountability should therefore both be provided. Within the study on the WASH programmes, however, there was a lack of indicators in every project that allowed for the equity of outputs of outcomes to be monitored. Although this is included within the methodology for testing VfM for DFID, work still needs to be done on implementing this on the ground with groups and services responsible for delivery the programmes. There has been, and still is, a large debate around these conceptual issues within the field of development studies. In 2010 The Big Pushback was created which rejected the narrowing of how value is measured within the sector. This has now been formulated into The Big Pushforward which is a group of practitioners who discuss and campaign for a mixed methods approach to measuring value, which recognises that both politics and power can be involved in the use of analyses such as VfM. The group works on developing methods of measuring value in development programmes which recognises that development works on the premise of building relationships and delivering social justice, and this cannot easily be measured by quantitative metrics which put a financial value on all outcomes.
DFID
Upward accountability
Downward accountability
Yes - big emphasis on upward accountability to funders including tax payers due to nature of project
Yes – emphasis on measuring actual impacts and outcomes
Who?
Wide range of international stakeholders ranging from a variety of governments, to NGOs
Individuals and communities in receipt of assistance are included within approach
Whose values?
Looks at cost-
Inclusion of equity shows 20 | P a g e
How?
efficiency and costeffectiveness alongside ‘3 Es’
recognition of individual/community values
Measurement of financial indicators such as costs, capital, and labour
Measurement of sustained actual outcomes and impacts
Approach from Health Sector (QALYs)
A sector which also faces trouble in defining and analysing VfM is the health sector. This section will focus on one tool of measurement (QUALY) which links to the previous discussion around the measurement of sustained actual outcomes for individuals. There are a variety of approaches taken within the health sector, with a constant drive towards efficiencies and looking at VfM. The QALY is focussed on as it links to the inclusion of ‘equity’ from the DFID report as well as the focus on tenant inclusion from HouseMark reports. When looking at VfM within the health sector, Smith (2009) outlines that there are two fundamental economic concepts: allocative efficiency and technical efficiency. The first indicates the extent to which funding which is limited is used to purchase the appropriate mix of services for those who provide the funding. In this way, funders can see that their input maximises health gains for service users. The second relates to the extent to which the intervention can be delivered for a minimum cost but maximum quality of output. Smith emphasises that “when deciding what services to purchase the main (but not sole) focus of allocative efficiency is prospective” (2009, p. 6). There are a number of ways by which to do this. NICE outline that they have shifted from a focus in cost-utility up until 2012, to a focus on cost-consequence and cost-benefit analysis. Within this process they use the tool of Quality Adjusted Life Years to measure the effectiveness of a public health intervention. This tool is defined as “the measure of the health of a person or group in which the benefits in terms of length of life are adjusted to reflect the quality of life”. In this analysis, one QALY is equivalent to one year of perfect health. This is calculated by estimating the years of life that a patient has remaining following a particular intervention, and weighting that against a score of quality of life within those years. This score is measured in terms of individuals’ ability to carry out day to day activities, free from physical pain or mental disturbance. NICE generally considers that interventions costing the NHS £20,000 or less per one QALY are cost-effective. The expansion of analysis from utility to benefit and consequence means that this measurement is now one of a range, where others look at indirect impacts such as how the 21 | P a g e
individual functions within broader society, and intangible impacts such as quality of life and living with pain. Collectively, these give a ‘balance sheet’ of outcomes which decision makers can use in relation to the cost of an intervention. Although NICE puts a financial value on a QALY, there is still a link between DFID’s emphasis on sustained actual outcomes and a measurement of quality of life. This can be understood as a focus on functionings, or what people can do or be. The QALY measures individuals’ ability to undertake daily activities without pain or disturbance and equates this to the cost of intervention. DFID emphasise equity and sustainability in the ability of individuals to function within their measurement framework. Richard Cookson (2005) has expanded on this focus on doing and being in the context of developing a ‘capability QALY’ which expands the health measurement to provide an analysis of individuals’ functionings. This, he argues, “acknowledges the non-separability of health and non-health components of wellbeing” (2005, p. 827) which can be seen to be reflected in NICE’s shift from measurement of utility to benefit and consequence. Cookson describes this measurement as “a cardinal and interpersonally comparable index of the value of an individuals’ capability set in a given time period under certainty” (2005, p. 818). The structure of the measurement is used in the same way, however it focusses on wellbeing more generally rather than health in particular. The capability QALY could be used to provide that ‘balance sheet’ of outcomes that NICE calculates through a variety of measurements. Capability QALY
Who?
Whose values?
How?
Upward accountability Yes - demonstrates the impact of a specific amount of funding to organisation (NHS) or taxpayer Wide range of stakeholders; government, NHS, specific institutions, taxpayers Financial aspect focuses on funding bodies Relationship between financial measure and measure of individuals’
Downward accountability Yes - takes account of individuals’ ability to do and be things related to the use of resources Measurement focuses on outcome for patient
Quality aspect focuses on individuals’ abilities Relationship between financial measure and measure of individuals’ 22 | P a g e
ability to live well
ability to live well
KEY POINTS
As previously noted, these two examples of equity and the use of QALY can be seen to link to the inclusion of tenants in the previous section. It could be argued that all of these measurements are able to provide both upward and downward accountability and focus on tangible impacts for individuals who are receiving services.
1. Within the divergence of VfM models across the public sector some focus on upward accountability and some focus on downward accountability 2. The Development Sector includes the idea of equity in its measurement - this is the recognition of sustainable outcomes for people's lives 3. The health sector uses the idea of Quality Adjusted Life Years as one measure of VfM - this looks at individuals' quality of life following health spending 4. Both of these approaches are person-centred, and can provide an insight into a tenantfocussed approach to VfM in Wales
23 | P a g e
Value for Money Literature Review September 2016
4.0 International Models This section of the review will look into international models of value for
Introduction money, focussing specifically on The Netherlands and Australia. The Netherlands is of importance to the subject because historically social housing was moved away from government regulation so this chapter includes a review of the minimal literature produced for approaches taken to maximising value for money at a time when there was a particular need for a market oriented approach. The Netherlands has since returned to regulation arrangements and this chapter will also look into more recent research undertaken on the subjects of a ‘systems approach’ and the correlation between scale and value for money. In Australia, extensive research has been undertaken to inform the measurement of social landlord performance, and in particular, the strengths
and
weaknesses
of
existing
measures
of
housing
management inputs and service outcomes, a determination of how management expenditure per dwelling should be defined and measured, an exploration into how landlords should maximise added value on wellbeing outcomes, how added value should be effectively quantified and measured and finally, how existing methods and measures adopted in Australia should be adapted to promote a comparison across providers.
The Netherlands The social housing sector in the Netherlands is unique in terms of its structure and size. Ziljstra and Van Bortel (2014) explain that 2.4 million social housing properties are managed which equates to 32% of the total housing stock, making the social housing sector in the Netherlands the largest in Europe. The U.K is the second largest with 20% of the stock.
24 | P a g e
Portfolio Analysis
As a result of Dutch social landlords “…helps the analyst operating largely independently from to find the the government in the 1990’s, portfolio analysis, an approach commonly combination of associated with the private sector, assets with the was adopted. Van der Flier and Gruis maximum return for (2002) discuss how the basic assumption of portfolio analysis is that any given degree of “the prospects of a firm’s strategic risk.” business areas (SBA’s) are determined by the growth of the demand for it on the one hand and by the competitive strength of the firm on the other.” Portfolio analyses can be an effective tool to support decision making and determine the appropriate allocation of resources to the various SBA’s of an organisation. The authors discuss an approach to “The main portfolio analysis known as the Growth Share Matrix which was advantage of the developed by the Boston Consulting BCG matrix is its Group (BCG). One axis of the BCG simplicity: there are matrix plots volume growth in demand as a measure for future only four prospects. On the other axis they plot possibilities and the the firm’s market share as a measure prescriptions are of its future competitive position… New products need a lot of (R&D) clear-cut.” investment to develop. However, the returns are small because of their low market share. The resulting cash flow is negative. The products are called ‘question marks’. When the product is successful the market share increases and the cash flow becomes zero: ‘stars’. After some time the market matures, the R&D 25 | P a g e
costs have been recouped and the cash flow becomes positive ‘cash cows’. In the last part of the cycle the produce becomes obsolete. The market and the market share decreases, and the cash flow becomes negative: ‘dogs’. The BCG diagram informs decisions on future participation in respective Strategic Business Areas (SBA’s). The ‘stars’ should be cherished, the ‘dogs’ should be divested, the ‘cash cows’ should send their excess cash to the headquarters to use in other SBA’s and the ‘question marks’ should be analysed to see whether the investment into converting them into stars is worthwhile. (Van der Flier and Gruis, 2002)
The authors also discuss the differences in the way landlords can a portfolio manage their stock resulting largely analysis approach from different business objectives. associations in the be effectively Housing Netherlands, as in the U.K invest in adopted by social their stock on a non-profit basis – landlords?” “often with a price and quality that would not be realised by commercial landlords.”
“The question is,
The usefulness of portfolio analysis within the social housing sector comes into question due to the “absence of a profit motive” as the “emphasis in these analyses lies on 26 | P a g e
financial performance.” The authors explain that although financial performance had become more important to social housing providers in the Netherlands at this time due to their financial independence, it was “not central to their management decisions.” Van der Flier and Gruis (2002) argue that portfolio analysis can be effectively adopted by social landlords in two ways: 1. To analyse the market prospects of the dwellings in their portfolio, using market demand and future competitive strength as the dimensions of the matrix. Depending on the mission statement, a social landlord will focus on certain parts of the market demand the demand of the target group(s). The return on investment can be represented by several measures, such as the internal rate of return (IRR) or overall rate of return (ORR) or as the net present value (NPV). In all these measures, the return is calculated on the basis of the future cash-flow, derived from the investment. For a measure of return on existing stock, we argue the NPV is preferable. 2. To analyse the going concern value in relation to the market value. As the going-concern value reflects what a landlord is expecting to earn with its dwellings under current policy, the market value reflects the maximum financial revenues a social landlord can realise with its dwellings (for example by maximising rents and sale). The difference between the goingconcern value and the market value can be seen as economic (opportunity) loss, which can be accepted if a social landlord deems it necessary to realise their goals in social housing. (Gruis 2001; Van der Flier and Gruis 2002)
Conclusion
Van der Flier and Gruis (2002) conclude that although portfolio analyses are compatible with the objectives of housing associations, 27 | P a g e
The systems Approach
KEY POINTS
there are restrictions in terms of the emphasis of financial performance associated with this approach, as discussed earlier. “For Dutch housing associations, which have to operate as financially independent not-forprofit landlords, this performance is becoming more important but is not conclusive for their management decisions.” The second restriction results from the limited possibilities at the time for housing associations to diversify, although as activities in a variety of sub-markets have increased for social housing providers, this approach can support decisions on what areas of diversification are most appropriate and viable. 1. Portfolio analysis can be useful for making investment decisions 2. The emphasis of this approach is on financial performance 3. It can be useful to analyse market prospects of dwellings (return on investment) 4. It can be useful to analyse the going-concern value of dwellings 5. It can help with decision about diversification
“Performance measurement is essential to make organisations function effectively and efficiently.”
Straub et al (2010) argue that performance measurement is not just of vital importance as a management instrument; it is indispensable for external control of public organisations. External supervision is greatly facilitated when outsiders have access to the same unequivocal and transparent information on the operations of the organisation as the management of the organisation itself.
Straub, Koopman and van Mossel undertook research in 2010 to identify the effectiveness of the systems approach to performance management in social enterprises, in particular the maintenance service delivery of Dutch housing associations and argue that as the main objectives of ‘not-for-profit organisation’ are of a ‘non-monetary nature’ performance measures should not consist of financial indicators alone.
28 | P a g e
“The concepts of service efficiency and effectiveness relate to an understanding of public-service delivery as a process which uses resources (inputs such as effort and time spent by the staff, use of buildings and equipment) to produce services (outputs) in order to achieve an objective (outcomes) (…) Service efficiency is the rate at which resources (inputs) are converted into services (outputs), while service effectiveness is the extent to which services provided actually achieve the intended objectives (outcomes).” (Straub et al, 2010)
Straub et al (2010) argue that whereas product measurement is in fact output measurement, process measurement implies the measurement of throughput at any given point in time during the operation. They state that output and throughput measurement are alternative ways of measuring performance, “in the sense that output (product) measurement reduces the multi-value performance to a “single-valued” one: the quantifiable part of the performance.” Importantly, the authors discuss how output measurement disregards the fact that performance results from co-production among many actors and thus ignores the interactions and synergies in the production process. “Throughput (process) measurement, on the other hand, includes the efforts of The production multiple actors with multiple goals, as well as co-productions.”
process in a “The conceptual systems approach considers organisations or parts of an organisation as separate systems, operating in an environment.” nutshell (De Leeuw, 1990). The illustration below supports the argument of Straub et al who discuss how the system and the environment are “defined simultaneously” with the system being “distinguished from its environment by a clearly discernible border.
The authors explain how the system interacts with its environment through the exchange of materials, energy and/or information. The incoming interactions are called input, while the outgoing interactions 29 | P a g e
are dubbed output. The transit substance is labelled throughput, while
A system and its the actual effect of the output on the environment is called outcome. environment The combined use of output, outcome and throughput measurement enables the assessment of multitask performance fields of public organisations. The input and output indicators define the beginning and ending of the production process that finally generates the outcome, while the throughput indicators are defined somewhere during the production process. (Straub et al, 2010)
Straub et al discuss how although “human organisations” can be “complex and dynamic”, they can still exhibit “predictable and manageable behaviour”. The authors argue that performance management is achievable as long as there is an acceptance that contributions from separate actors within an organisation may not be “fully identified and the “output or the outcome of the operations are seen as stochastic quantities, subject to risk and uncertainty, rather than the deterministic quantities of the systems approach.” The authors argue that an organisation “can be considered as a black box, yet with clear causal relationships with the actors and objects in its environment.” Stewart and Ayres (2001), cited in Straub et al (2010) argue that the “nature of a problem cannot be understood without referring to possible solutions for the problem.” The authors discuss how the use of systems concepts can help to “rationalise a given problem, even if they fail to give an exact description of the problem or the definitive solution.” They found that performance management could still be an effective managerial tool for evaluating existing practices and recommending future policy direction.” Using systems thinking or better yet, the systems approach in complexity thinking, performance measurement and steering becomes a practical exercise, rather than a theoretical one.” (Straub et al, 2010) Koopman et al (2008) argue that a housing association can be described as a super-system engaged in feed forward steering of the environment. Straub et al (2010) explain that housing association (super-system) can be divided into separate parts based on the 30 | P a g e
managerial layers of the organisation. They argue that this division will usually consist of the managerial level (the meta-system) and the various operational level of the association. For input measurement, indicators will usually consist of time and money spent on operations. The choice of performance indicators for the outcome measurement and subsequently the appropriate output and throughput measurement should be derived from the public tasks of the housing association. The external goals define the outcome. In the Netherlands, as in the U.K, housing association’s goals are defined individually however, they must fit within one of the following criteria as prescribed in The Social Housing Management Decree which is derived from the Dutch Housing Act: (1) Guarantee the financial continuity of the housing association. (2) Provide affordable housing to low-income tenants. (3) Maintain the quality of the housing stock. (4) Ensure tenant empowerment by giving tenants a say in policy matters and housing management. (5) Increase and maintain the quality of life in the area surrounding the dwellings. (6) Provide joint housing-and-care arrangement. (Ministerie van VROM, 2005)
It is the responsibility of the housing association to develop individual performance indicators which ensures the effectiveness and efficiency of business processes and provides accountability to external stakeholders, including local and central government, tenants and financial supervisors. Straub et al (2010) discuss how the systems approach has the tools to bring transparency to the aims and means of the various participants in the production process of housing associations. As soon as the aims and means are transparent they can be discussed and used for problem analysis. Given that housing associations have many complex aims and tasks both in the market and in the public domain, there is a pressing need for an analytical framework in which aims are transparently defined and insight is provided into the production process that leads to them. “The identification of key performance indicators, as well as the input, throughput, output and outcome indicators that are bound up with it, facilitates the choice, implementation and (re-)shaping of policy alternatives and the justification for choosing among these alternatives. Ideally, the set of performance indicators or key performance indicators (KPI’s) ensures the effectiveness and efficiency of the internal business 31 | P a g e
KEY POINTS
process and also covers the accountability to external stakeholders.” (Straub et al, 2010)
Does Size Matter?
1. Measures process (throughput) rather than output 2. Includes the efforts of many actors in performance (co-production) 3. The combined use of output, outcome and throughput measurement enables the assessment of multi-task performance 4. Systems concepts help to rationalise a given problem 5. The choice of performance indicator for the outcome measurement should be derived from the public tasks of the housing association - the external goals define the outcome
Following a period of fraud and mismanagement and accompanied by a period of government austerity measures (due to economic crisis in the Netherlands), and the widely held view that Dutch housing associations are inefficient, the Dutch Parliament started an enquiry to determine whether increasing the scale of housing associations in recent years has contributed to their ability to deliver value for money. Zijlstra and van Bortel (2014) asked the question: has the economic crisis and the housing market downturn generated new insights in the performance and resilience of housing associations? In their analysis the authors explored the relationship between performance and organisational scale using the following indicators:
1. 2. 3. 4.
Operational & maintenance costs per managed unit Average number of managed units per employee Costs per employee Activity level of the organisation
Tenant satisfaction was used as the outcome of the association’s performance.
The table below outlines the analytical framework with aspects used as performance indicators and aspects used as indicators for expected resilience: 32 | P a g e
Organisation:
Performance Operational costs Maintenance costs
Employees/dwelling
Euro’s/dwelling Euro’s/employee Activity level Managed priorities/employee Organisational culture:
Results and outcomes
Resilience Opportunities savings on operational costs Opportunities savings maintenance costs (maintenance expenditures have not explicitly been asked in the questionnaire) Opportunities for downsizing the amount of staff members … Effect on housing production … Service delivery components (additional services) Aim on digitisation
Tasks interpretation (core tasks or broad ‘societal’ tasks) Cooperation with ‘partners’ Tenant satisfaction Quality of service delivery Future handling of tenant participation (Zijlstra and van Bortel, 2014)
Following the deregulation of the social housing sector in the Netherlands and the resulting financial and operational autonomy in the 1990’s, mergers became a key feature with the average number of dwellings per association increasing from around 3,000 in 1997 to 6,300 in 2012. Zijlstra and van Bortel (2014) discuss how this was as a result of support for the argument that merging organisation's would bring economies of scale, reduced operating costs and increased effectiveness “in the field of housing construction, refurbishment and neighbourhood renewal” however, van Bortel, Mullins and Gruis found 33 | P a g e
no relation between scale and performance’ in their 2009 case study referring to social landlords in England and the Netherlands, as cited in Zijlstra and van Bortel (2014) The authors argue that the results of the case study do demonstrate “that growth brings with it significant opportunities for scale-related efficiencies, but scale does not automatically lead to better value for money. Organisational transformation is necessary in order to benefit from efficiencies as the opportunities arise. Scale might be important if, but only if, an organisation can make the necessary changes in its culture and business processes to achieve any economies of scale. The case studies findings indicated that housing association’s staff is the primary vehicle through which “Housing efficiencies are delivered. association stock Communication with staff and linking characteristics, staff expectations with specific outcomes are essential.”
financial resources The authors further explain how and local housing customer satisfaction data from KWH, market challenges the Quality Centre for Dutch Housing are important Associations has demonstrated that customers
of
large
housing
explanatory associations are less satisfied with factors.” landlord services than those of small organisations. “When we account for the variance within the size categories, we can conclude that small organisations (<1,800 units) “Some claim that have a significantly higher customer satisfaction than large housing higher operating associations (>10,000 units).” (Zijlstra costs are the result and van Bortel, 2014)
of a higher activity
“The survey indicated that most level.” housing associations expect that the reduction of operating costs will affect the nature of service delivery, but not the satisfaction of customers. They expect that making landlord services more standard and deliver these services more efficiently will reduce costs and will create more room for tailored services in exceptional cases. In other words: a mixed approach with standardised business process were possible and tailor made services were needed.” (Zijlstra and van Bortel, 2014)
34 | P a g e
KEY POINTS
In conclusion, the authors identified that “the bigger the association, the more contact moments they have with tenants, and thus the bigger efficiency advantages can be reached in several service actions. The other aspects of landlord services and online service delivery do not differ significantly between small and large associations.”
1. Research has found no relationship between the scale of a housing association and performance 2. Growth does bring significant opportunities for scale-related efficiencies, but not automatically better value for money 3. Housing association staff are the primary vehicle through which efficiencies are delivered 4. Smaller organisations (-1800) have significantly higher customer satisfaction than larger organisation (+10,000) 5. Housing associations expect that making landlord services more standard and delivering them more efficiently will reduce costs 6. Bigger associations have more contact moments with tenants and, therefore, bigger efficiency advantages can be reached
As Pawson et al (2014) explain, the restructuring of Australia’s social Australia housing to promote a more diversified mix of housing service providers has been accelerating in recent years. In 2012, 81 per cent of homes were being managed by eight state or territory government organisations, down from 88 per cent in 2004. A continuing process of diversification has resulted from two main strategies: transfer of the management (or, in some cases ownership) of existing public housing from state government to community housing providers (CHP’s) (documented in Pawson et al, 2013); and channelling government and private investment for new housing supply to larger CHP’s. (Pawson et al, 2014)
35 | P a g e
Pawson et al (2014) argue that ‘transparency’ and ‘accountability’ are
Assessing the key drivers from government in the expenditure of public funds, or Management Costs to put it another way, the effective use of resources. A broader & Tenant Outcomes motivation is the adage ‘if it isn't measured, it isn't managed’. This alludes to the value of statistical reporting obligations as an incentive
in Social Housing: for performance improvement of the measured service. Developing a Framework
Pawson et al undertook research into performance management in social housing in Australia in an attempt to answer the following questions: 1. What are the strengths and weaknesses of existing official measures of housing management inputs and service outcomes? 2. How should management expenditure per dwelling be defined, measured and disaggregated for application to a multi-provider system? 3. How do social landlords seek to maximise added value on wellbeing outcomes? 4. How can added value via tenancy management services be effectively quantified and measured? 5. How should existing assessment methods and measures of housing management service outcomes be adapted to promote comparison across provider entities and provider types?
The focus of the research outlined in this section of the literature review is based on question 1 – strengths and weaknesses of existing measurement techniques with a proposed measurement framework outlined at the end of the section.
36 | P a g e
Pawson et al (2014) explain that traditional social housing performance measures in Australia and elsewhere “tended to emphasise service efficiency measures such as the proportion of un-tenanted properties, the rate of stock under-utilisation (also termed under-occupation) or the incidence of rent arrears (uncollected rent as a proportion of the rent roll). However, the key ‘efficiency’ measure of unit costs designated within Australia’s official social housing performance indicators has been the net recurrent cost per dwelling measure.” (Productivity Commission 2013). The authors identified a reliance on tenant satisfaction metrics drawn from ‘custom-designed consumer surveys’ in Australia as in the UK, the Netherlands and elsewhere with a ‘national centralised survey’ providing the evidence in Australia. Housing associations have also undertaken their own tenant satisfaction surveys with resulting satisfaction rates being treated as important indicators of service effectiveness. Pawson et al also explore the significance of tenancy sustainment as an additional outcome-related performance indicator. “In contrast to tenant satisfaction metrics, tenancy sustainment measures are derived from analysis of administrative staff rather than being collected through social surveys. Beyond the measurement of tenancy sustainment, efforts have been made to quantify the ‘non-shelter outcomes’ of social housing – or the ‘added value’ of a social rental tenancy. In an Australian study focussing on such welfare impacts of being accommodate in public housing, it was found that recently housed tenants tended to have consequentially enjoyed improved health and better engagement with education for their children (Phibbs, 2005). In a similar vein, more recent work has attempted to apply the concept of social return on investment (SROI) within the social housing context.” (Pawson et al, 2014)
37 | P a g e
In terms of a current performance management framework for Australian social housing, an approach which was developed in the 1990’s is outlined below:
Social housing performance indicator framework
“The net recurrent cost per dwelling input expenditure measure comprises of four cost categories: 1. Administration costs – the cost of the administration offices of the property manager and tenancy manager 2. Operating costs – the costs of maintaining the operation of a dwelling, including repairs and maintenance, rates, the cost of disposals, market rent paid and interest expenses 3. Depreciation costs 4. The user cost of capital – the cost of the funds tied up in the capital used to provide social housing” (Pawson et al, 2014)
Pawson et al (2014) argue that disaggregating the measure into its component parts does not assist in prompting explanations for the drivers of cost differentials and/or with advising possible benchmarks for assessment of performance… It can be argued that a composite measure of social landlord costs of provision will be significantly 38 | P a g e
influenced by difference in asset profiles as well as by differences in landlord efficiency and the scope of management tasks being performed and likely in turn to be partly related to provider differences in client profile. “Our review of the net recurrent costs indicator suggests that a new more narrowly conceived measure of landlord expenditure would be required to analyse social housing landlord efficiency and to probe how effectively different types of landlords shape service outcomes. A better developed and negotiated definition of what comprises management tasks would also be required. This could usefully separate tenancy/property management tasks and welfare tasks to cater both for the distinctive nature of social landlordism and provider differences in client profiles, as well as enabling benchmarking against private landlords whose focus is solely on tenancy/property management.” (Pawson et al, 2014)
The research identifies a number of important strengths with the existing performance measurement approach. In particular the authors discuss the generation of “intelligible metrics which – thanks to the common methodology inherent in a centralised survey – can be reliably compared across jurisdictions and provider types. This refers to the contrast with administratively generated performance statistics which are often compromised by inconsistent data recording practices across participating organisations.” There are however, a number of weaknesses with the current system too as Pawson et al (2014) explain, the focus on customer satisfaction raises concern over what some consider to be the ‘nebulous’ concept of ‘satisfaction.” The authors suggest that “while retaining a surveybased approach, there could be value in experimenting with different forms of question such as the ‘would you recommend?’ query. In addition, the authors acknowledge the strength in a “cross –sectional coverage of the entire (social) population” however, argue that questions about service quality “could be more meaningfully directed only to those having recently received the service concerned. For example, while all respondents are asked their opinion of day to day repairs performance, many may have no direct experience of this service in the recent past (or ever in the case of newly housed tenants).” The research also raises concerns about the limitations of a ‘surveybased approach to performance management’ which according to the authors, requires ‘professional survey management’. In addition, the authors outlined the following potential limitations: 39 | P a g e
-
Does not capture social inclusion outcomes No information about property occupied by each respondent Does not measure actual as opposed to perceived Does not consider what would have occurred in the absence of provider assistance Relationship between respondent age an expressed satisfaction Property type – houses are more popular than flats (skews results) ‘The honeymoon effect’ newer tenants are generally more satisfied
Pawson et al (2014) discuss three approaches to the economic evaluation of social housing and consider the most appropriate to be cost consequences analysis (CCA). Approach Cost benefit analysis Cost effectiveness analysis Cost consequences analysis
Measure for each provider Ratio of housing costs to value of housing benefits Housing cost per tenant year Disaggregated housing costs and tenant outcome measures
As the authors explain, “this is partly because, unlike cost effectiveness analysis, CCA does not call for outcomes to be reduced to a single measure, in a complex service like social housing management we regard this as unrealistic. Indeed, we are looking at a housing system where variance in outcomes is likely to be considerable as different housing providers approach the task of supporting tenants in a variety of ways. The project is very interested in examining this variance in outcomes and not reducing to a single measure such as tenant years. Indeed a cost-effectiveness approach would limit the ability of the outcomes of the study to provide learnings for the social housing sector. Providers with higher costs per tenant’s years would simply argue that they have higher costs because their tenant-years are ‘better’.” The authors also argue that CCA is more appropriate that cost-benefit analysis because “unlike the latter, it does not require the assignment of financial values to all outcomes – something which would be difficult to operationalise in this context.”
40 | P a g e
Conceptual framework for measuring social housing cost of provision and tenant outcomes
The above diagram is a conceptual framework for measuring both social housing costs and tenant outcomes which has been developed by Pawson et al throughout their research. They explain that, “in the first column (housing management activities, disaggregated), we list some typical housing management tasks or components of the landlord role. The relative importance of these to individual providers will vary somewhat, depending on dwelling stock configuration and other matters. These activities are aggregated into the broad categories (management fields) proposed in the second (central) column. These categories for our proposed framework for the recording of housing management expenditure. It is envisaged that a common set of accounting protocols would be developed for disaggregating housing provider expenditure on salary costs (and outsourced contributions to service delivery).” (Pawson et al, 2014)
The authors also explain that “the first step in our proposed approach would be for state housing authorities and ‘in scope’ CHPs (proposed as those managing more than 1000 dwellings in 2014) to identify the salary expenditure associated with all staff with a role in providing some element of housing management services – defined as landlord activities other than: 41 | P a g e
-
Maintenance works implementation (rather than ordering supervision and reporting) Capital investment planning for and project management of stock reconfiguration and renewal.
-
By definition, landlord activities also exclude such things as the planning and financing of new build housing, assisting people to access private rental housing and any non-housing business activities.”
KEY POINTS
“In our view, such metrics could provide a means of usefully comparing the resource inputs to housing management activities across provider types and entities and, potentially, a basis for cost consequences analysis by relating unit expenditure on specific aspects of housing management to quantified tenant outcomes.” (Pawson et al, 2014)
1. Existing performance measures in Australia have included tenancy sustainment and added value measures in addition to tenant satisfaction 2. Social landlord costs of provision can be influenced by differences in asset profiles, landlord efficiency and scope of management tasks and, in turn, partly related to client profile 3. Customer satisfaction can be considered to be a nebulous concept of satisfaction 4. Survey questions about service delivery could be directed only to those who have received the service concerned 5. Survey approaches demand professional survey management 6. The researchers identified cost consequences analysis to be the most appropriate economic evaluation 7. The preferred framework for measurement is one that relates unit expenditure to quantified tenant outcomes
42 | P a g e
In the second part of the research undertaken, Pawson et al (2015)
Assessing attempt to answer the remaining four questions, starting with how Management Costs management expenditure per dwelling should be defined and disaggregated. & Tenant Outcomes in Social Housing: Within the authors proposed framework for quantifying social housing
management expenditure ‘housing management’ is defined as focusing
Recommended on ‘landlord services’ net of repairs and maintenance works Methods & Future expenditure. They argue that “this is a ‘purer’ measure of the ongoing Directions year to year resource cost of running a social housing portfolio. This is
because it excludes the potentially distorting impacts of differences in repairs and maintenance expenditure needs attributable to the original design of buildings, to construction materials, or to historic maintenance inputs, and other variable operating costs that are beyond the control of landlords (e.g. insurance and property rates).” The authors continue to explain that within the ‘housing management’ domain we advocate the separate identification of expenditure associated with four distinct elements of the overall management task, namely: tenancy management, property management, individual tenant support (ITS) and additional tenant and community services (ATCS). This could facilitate meaningful benchmarking of social and private landlord management expenditure by enabling the exclusion from such a comparison of the housing management activities (ITS and ATCS) specific to social landlords. In response to research question 3, the authors discuss approaches to maximising added value on wellbeing outcomes: “As this is reportedly an increasing cost pressure for all social landlords, having more robust appraisals of needs assessment approaches (such as those implemented through interviews and home visits) and support service models (e.g. case planning and referral processes) would be highly beneficial, along with having better outcome measures as discussed below.” “Within the social housing sphere there is a growing sense that inherent within the social landlord role is the promotion of longer term, ‘non-shelter’ opportunities and outcomes for tenants. There is growing emphasis on enhancing tenant employability and capacity to move to an alternative tenure, such as by renting privately or contributing equity to their housing (e.g. via various shared ownership models). Arguably, this remains to be explicitly stated by Governments and regulators. Nonetheless, while social landlords (especially CHPs) appear willing to pursue this goal, the means of achieving it seem to be in their infancy (from our case studies). Furthermore, in a highly resource-constrained environment, it is unclear how such additional services can or should be funded.” (Pawson et al 2015) In terms of how added value services can be effectively quantified and measured (question 4), Pawson et al discuss the option of using survey 43 | P a g e
instruments more effectively, such as tenancy sustainment rates for at risk households. The research also explored the feasibility of measuring the outcomes of social housing provider assistance in reconnecting work-capable tenants with employment. “In practice, however, the definition and operationalisation of such a measure is highly complex. In the view of the research team, such a measure cannot be feasibly derived from social landlords’ internal record systems.” (Pawson et al 2015) In response to question 5, the authors provide their views on how assessment methods and measures of housing management service outcomes can be adapted to promote comparison across providers. “We advocate some modest enhancements to existing survey-based measures of housing management service outcomes. In particular, incorporating an increased emphasis on recently housed tenants would enable the National Social Housing Survey to delve deeper into how effectively social landlords assist new tenants in terms of social inclusion and economic reconnection. Within the realm of administratively-generated performance metrics, it would be desirable to extend to larger CHPs the current obligation to report on tenancy sustainment rates. Initially, this could focus on ‘greatest need’ new tenants—consistent with the current framework. Ideally, the remit of this metric would be narrowed for both types of providers to focus more specifically on ‘at risk’ tenants—such as those formerly homeless at the point of rehousing. Agreement on a precise definition of ‘at risk’ tenants would need to be negotiated directly by the state/territory governments and community housing industry bodies.” (Pawson et al, 2015) The authors also express the importance of using cost-consequence analysis (CCA) in the process of comparing organisations as the CCA model allows for multiple outcomes to be documented without the requirement for a monetary value to be attributed (as in cost-benefit analysis). Interpretation can then allow for differences in costs and performance to be explained in relation to the operating environment (e.g. explaining differences by operating scale or by geographic factors, such as remoteness or stock sparsity).
44 | P a g e
KEY POINTS
1. When quantifying social housing management expenditure, housing management is defined as focusing on landlord services, net of repairs and maintenance works expenditure 2. Housing management should include four distinct elements – tenancy management, property management, tenants support and tenant and community services in order to facilitate meaningful benchmarking 3. To effectively measure added value services, survey instruments such as tenancy sustainment rates for at risk households could be used, in addition to the measurement of outcomes of reconnecting work-capable tenants with employment 4. An increased emphasis on surveying recently housed tenants would identify how effectively new tenants are assisted in terms of social inclusion and economic reconnection
45 | P a g e
Value for Money Literature Review September 2016
5.0 Social Value
There are many definitions of social value with varying degrees of The Definition(s) similarities and differences. Some definitions are focussed on procurement activities and the ‘added value’ that can be derived from of Social Value the awarding of contracts. Other definitions are structured around wellbeing activities in an attempt to measure the ‘softer outcomes’ of community development and other frontline housing services. When considering value for money, social value can be an effective approach to measuring the non-financial outcomes of housing associations, as discussed earlier in the literature review. Social value is the value that stakeholders experience through changes in their lives. Some, but not all of this is captured in market prices. (Social Value UK, 2016)
Social value is a way of thinking about how scarce resources are allocated and used. It involves looking beyond the price of each individual contract and looking at what the collective benefit to a community is when awarding a contract. (Social Value Act 2012)
Social value is the relative importance of changes that occur to stakeholders as a result of an activity. (SROI Network, 2016) The many different definitions of social value may be contributing to the confusion around the subject in the social housing sector. As Wood and Leighton (2010) discuss, “there are several, and diverse, methods for measuring social value, and this fragmentation may be a factor in the poor penetration of social value reporting in the third, statutory and commercial sectors.
46 | P a g e
KEY POINTS
1. Social value can relate to procurement and wellbeing activities 2. Social value can be an effective measurement tool 3. Social value can also be a powerful planning tool to inform decision making 4. The many different definitions of social value could be contributing to confusion around the subject
Measuring social value is about measuring softer outcomes however, Monetary vs Non- the introduction of social return on investment (SROI) has taken this monetary one step further as it has assigned monetary values to outcomes, either anticipated, or achieved. The New Economics Foundation (2016) outline the concerns with traditional cost-benefit analysis approaches not considering anything “anything beyond simple cost and price.” The SROI approach measures a “much broader concept of value, taking into account social, economic and environmental factors.” £1 invested in high-quality residential care for children generates a social return of between £4 and £6.10. £1 invested in alternative, non-prison based sentencing for women offenders generates a social return of £14. (New Economics Foundation (NEF), 2016) Potential SROI Benefits It can help organisations understand what social value an activity creates in a robust and rigorous way and so manage its activities and relationships to maximise that value The process opens up a dialogue with stakeholders, helping to assess the degree to which activities are meeting their needs and expectations
Potential SROI Limitations If there are not already good outcomes data collection systems in place, it can be timeconsuming to conduct an evaluative SROI analysis first time around There is a danger of focusing narrowly on the ratio. The ratio is only meaningful within the wider narrative about the organisations. Just as an astute investor would not make a financial decision based on just one number, the same practice applies to this social measurement tool. For this reason, comparisons between 47 | P a g e
organisations just based on the ratio are not recommended SROI is an outcome, rather than a process evaluation. The dialogue with stakeholders yields some insight into what works and what doesn’t and why, but there may be instances where a more specific process evaluation would be useful
SROI puts social impact into the language of 'return on investment’, which is widely understood by investors, commissioners and lenders. There is increasing interest in SROI as a way to demonstrate or measure the social value of investment, beyond the standard financial measurement Where it is not being used SROI requires a diverse skill set already, SROI may be helpful in – from stakeholder engagement showing potential customers (for to working with Excel example, public bodies or other spreadsheets. This can be hard large purchasers) that they can to find in one person develop new ways to define what they want out of contracts, by taking account of social and environmental impacts SROI can also be used in strategic management. The monetised indicators can help management analyse what might happen if they change their strategy, as well as allow them to evaluate the suitability of that strategy to generating social returns, or whether there may be better means of using their resources New Economics Foundation (NEF), 2016) Attaching a monetary value can be a useful way of measuring outcomes, either projected or achieved and can provide quantitative data to inform the most cost effective investment decisions however, there are also some risks of using this approach. Wood and Leighton (2010) discuss NEF’s evaluation of two rounds of SROI in 2008 in which it worked with several social enterprises to carry out an SROI analysis of its work. The subsequent report made a number of important findings: although organisations participating in the study found the process of SROI analysis a useful one, many found it challenging to collect all of the data they needed – often the data was outcome rather than output based, and monitoring and follow-up 48 | P a g e
processes were simply not in place in most organisations to collect this sort of information. Many organisation had to try to collect data retrospectively and found SROI time consuming and resource intensive, with few participants able to spare the staff to carry out the tasks required. Many also found the methodology and concepts hard to follow. As a result, there was a wide variation in the SROI ratios, as some organisations provided incomplete or incorrect data or did not follow the process properly.
KEY POINTS
Tuan (2008) discusses the subject of ‘silver bulletism’ which is the theory that there is ‘one special number’ that tells an organisation whether it is succeeding or failing, driven by the “desire to define a ‘bottom line’ that will do for the philanthropy and public sector what profit/loss statements do for the private sector.”
1. A number of models of measuring social value assign a monetary value to outcomes 2. Attaching a monetary value can provide useful quantitative data 3. This approach can be challenging in terms of data collection 4. This approach can be time and resource intensive
Models of Social Value
In Wales there are a number of methods of calculating and measuring social value used throughout the social housing sector. Outlined below are descriptions of the most widely used methods: Social Return on Investment (in addition to the information above): Social Return on Investment (SROI) is a method for measuring and communicating a broad concept of value that incorporates social, environmental and economic impacts. It is a way of accounting for the value created by our activities and the contributions that made that activity possible. It is also the story of the change affected by our activities, told from the perspective of our stakeholders SROI can encompass all types of outcomes – social, economic and environmental – but it is based on involving stakeholders in determining which outcomes are relevant. There are two types of SROI:Evaluative SROIs are conducted retrospectively and are based on 49 | P a g e
outcomes that have already taken place. Forecast SROIs predict how much social value will be created if the activities meet their intended outcomes. SROI was developed from social accounting and cost benefit analysis, and has a lot in common with other outcomes approaches. However, SROI is distinct from other approaches in that it places a monetary value on outcomes, so that they can be added up and compared with the investment made. (Social Impact Scotland, 2016)
HACT Wellbeing Valuation Toolkit: Wellbeing Valuation allows you to measure the success of a social intervention by how much it increases people’s wellbeing. To do this, the results of large national surveys are analysed to isolate the effect of a particular factor on a person’s wellbeing. Analysis then reveals the equivalent amount of money needed to increase someone’s wellbeing by the same amount. The main advantage of Wellbeing Valuation is that the values are consistent and robust. The consistency means that while you may be examining values for different types of outcomes, you are still comparing like with like. Wellbeing Valuation is in HM Treasury’s Green Book – the UK Government’s core guide to policy evaluation – as a method for placing values on things that do not have a market value through being bought and sold. (HACT, 2016)
Centre for Regeneration Excellence in Wales (CREW) – Landscape, Atmosphere and Horizon: This toolkit has been designed to provide a detailed understanding of life in a community. You might use it to establish a baseline measurement before you implement a programme of regeneration or you can use it to assess the impact of recent regeneration activity. Understanding the conditions in a locality before interventions commence is essential if you are to fully understand what effect policies and programmes are having. Regular monitoring of programmes as they are delivered is equally essential if they are to remain on track and meet intended objectives. Finally, an assessment of the impact of a programme when it ends will allow judgements to be made of its success or failure which can inform future delivery and also encourages the development of good practice. This toolkit combines statistical sources with local knowledge to develop a comprehensive understanding of what is happening at community level. (Centre for Regeneration Excellence in Wales, 2012)
50 | P a g e
Value Wales Toolkit: Value Wales is responsible for shaping policy, monitoring practice, supporting and advising professionals, developing the procurement profession, and compliance with EU regulations. The Wales Procurement Policy Statement sets out the procurement practices and actions required of all public sector organisations in Wales. Value Wales helps the Welsh public sector realise improved value for money through ‘smarter procurement’ by 1. Increasing savings through collaboration 2. Improving process efficiency especially through use of technology 3. Protecting the economy by encouraging smaller and more local suppliers and seeking re-investment in local communities 4. Building procurement capability Value Wales' Procurement Route Planner provides step-by-step guides to buying goods and securing services. It enables a consistent approach that mirrors the Welsh Government's 4-stage commissioning cycle. (Welsh Government, 2014)
Global Value Exchange: The Global Value Exchange is a crowd sourced database of Values, Outcomes, Indicators and Stakeholders. It provides a free platform for information to be shared enabling greater consistency and transparency in measuring social & environmental values. The site empowers users by giving them a voice to share their experiences and allow them to become the 'creators of knowledge'. The site is an exciting development because unlike anything else... It holds detailed information on Stakeholders, Outcomes, Indicators & Values - all together for the first time! All four sections of the site are inter-connected so you can see the relationships between entries. E.g. which outcomes are related to which stakeholders and which indicators are related to which outcomes or what value can be associated to that outcome. The Chain of Events feature allows you to develop theories of change and link outcomes to other outcomes. The Outcomes Matrix allows you to see which outcomes are being used by which organisation. The site has links to research that supports the information within each section. Users of the site are able to interact with the information. They have the ability to comment, rate and say how they use the existing entries. This crowd sourcing provides a form of validation and guidance for users. Users are able to add their own entries. Open Source. (Global Value Exchange, 2016)
51 | P a g e
Monetary Models of Social Value
Non- monetary Models of Social Value
52 | P a g e
KEY POINTS
1. SROI is a method of measuring and communicating a broad concept of value that incorporates social, environmental and economic impacts 2. There are two types of SROI: - Evaluating: retrospective analysis - Forecast: predictive analysis 3. The HACT Wellbeing Valuation Toolkit measures the success of an intervention by how much it increases people’s wellbeing 4. The HACT model values things that do not have a market value through being bought and sold 5. The CREW approach provides a detailed understanding of life in a community 6. The CREW toolkit can be used to establish a baseline measurement of to assess impact 7. The CREW toolkit combines statistical sources with local knowledge 8. The Value Wales toolkit helps the Welsh Public Sector realise improved value for money through smarter procurement 9. Global Value Exchange measures social and environmental values 10. The Global Value Exchange has developed a crowd sourced database of values, indicators and stakeholders 11. The chain of events feature allows users to develop theories of change
53 | P a g e
Value for Money Literature Review September 2016
6.0 Recommendations There are a number of key themes that have emerged from this Review. The main regards the difference between ‘upward’ and ‘downward’ accountability. The discussion of the different models and approaches across sectors shows the importance of considering this key question of accountability. In the housing context, are VfM measurements in place to providing assurance for service providers, service users, or both? The approaches proposed by HouseMark in both Scotland and Wales focus on the inclusion on tenants; ensuring a form of ‘downward’ accountability. This is echoed in approaches within International Development and Health which look at how VfM measurements can take account of the impact that investment in services can have on individuals’ lives. This approach could be built on in terms of creating a Welsh model for VfM for Housing Associations. This would enable the Welsh Government to take a ‘made in Wales’ approach to this concept, and align this with broader priorities. By focusing on tenants and what they are able to do and be, National Indicators related to the Wellbeing of Future Generations (Wales) Act can be met. In particular, this can be seen to map across to the Wellbeing Duty included in this piece of legislation. Recommendations for VfM in Wales: 1. Focus on achieving a balance between downward and upward accountability 2. Look at developing a tool which combines financial measures with an indicator of what tenants are able to do and be because of this investment 3. Take a co-regulation approach to developing a new model of VfM – including tenants 4. The Regulator to take an enabling rather than enforcing approach to VfM, thus supporting the co-regulation approach 54 | P a g e
7.0 Bibliography Alwardat, Benamraoui, and Rieple. 2015. Value for Money and Audit Practice in the UK Public Sector. International Journal of Auditing 19 (3), pp. 2016 – 217 Cabinet Office. 2012. Social Value Act: Information and Resources. Available at: https://www.gov.uk/government/publications/social-valueact-information-and-resources/social-value-act-information-andresources (Accessed: July 2016) Cookson, R. 2005. QALYs and the Capability Approach. Health Economics 14, pp. 817-829 Centre for Regeneration Excellence in Wales. 2012. The Atmosphere, Landscape and Horizon Regeneration Impact Toolkit. Available at: http://www.regenwales.org/resource_17_The-Atmosphere--Landscapeand-Horizon-Regeneration-Impact-Toolkit (Accessed: July 2016) Emmi, et al. 2011. Value for Money: Current Approaches and Evolving Debate. London: LSE. Glendinnig, R. 1988. The Concept of Value for Money. International Journal of Public Sector Management. 1 (1), pp. 42 – 50. Global Value Exchange. 2016. Discover Your Social Value. Available at: http://www.globalvaluexchange.org/ (Accessed July 2016) HACT. 2014. Measuring the Social Impact of Community Investment: A Guide to using the Wellbeing Valuation Approach. Available at: http://www.hact.org.uk/measuring-social-impact-communityinvestment-guide-using-wellbeing-valuation-approach (Accessed: July 2016) HCA. 2016. Regulating the Standards. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_d ata/file/535067/Regulating_the_Standards_July_2016.pdf. (Accessed: July 2016)
55 | P a g e
HCA. 2016. Delivering better value for money: understanding differences in unit costs – summary report. Available at: https://www.gov.uk/government/uploads/system/uploads/attachment_d ata/file/527847/Unit_cost_analysis_-_summary_report.pdf. (Accessed: July 2016). HouseMark. 2015. How do you know if you are providing value for money? Defining, managing, and demonstrating Value for Money in Scotland. Available at: https://www.housemarkbusinessintelligence.co.uk/Events_pdf/How%20 do%20you%20know%20if%20you%20are%20providing%20value%20f or%20money%20REPORT.pdf (Accessed: June 2016) HouseMark. (forthcoming). Defining, delivering, and demonstrating Value for Money in Wales. New Economic Foundation. 2016. Social Return on Investment. Available at: http://www.neweconomics.org/publications/entry/a-guideto-social-return-on-investment (Accessed: July 2016) Pawson, et al. 2014. Assessing Management Costs and Tenant Outcomes in Social Housing: Developing a Framework. AHURI Pawson, et al. 2015. Assessing Management Costs and Tenant Outcomes in Social Housing: Recommended Methods and Future Directions. AHURI Smith, P. 2009. Measuring value for money in healthcare: concepts and tools. London: The Health Foundation. Social Impact Scotland. 2016. What is SROI? Available at: http://www.socialimpactscotland.org.uk/understanding-socialimpact/methods-and-tools/sroi/what-is-sroi/ (Accessed July 2016) Social Value UK. 2016. What is Social Value? Available at: http://www.socialvalueuk.org/why-social-value/ (Accessed: July 2016) Straub, et al. 2010. Systems Approach and Performance Measurement by Social Enterprises. Emerald Insights. Tremolet, et al. 2015. Value for Money analysis of DFID-funded WASH programmes in six countries. Available at: http://vfm-wash.org/wpcontent/uploads/2015/08/OPM-2015-Synthesis-report-of-6-VFMWASH-studies.pdf (Accessed: June 2016) Trosa, S. and Williams, S. 1996. Benchmarking in Public Sector Management. Performance Management in Government: Contemporary Illustrations. Occasional Paper No. 9. Paris: OECD Tuan, M. 2008. Measuring and/or Estimating Social Value Creation: Insights into Eight Integrated Cost Approaches. 56 | P a g e
Van der Flier and Gruis. 2002. The Applicability of Portfolio Analysis in Social Management. Routledge. Welsh Government. 2014. Value Wales. Available at: http://gov.wales/topics/improvingservices/bettervfm/?lang=en (Accessed: July 2016) Wood, C and Leighton, D. 2010. Measuring Social Value: The Gap Between Policy and Practice. Zijlstra and van Bortel. 2014. Will Scale Finally Deliver? Delft University of Technology
57 | P a g e
