Afrobarometer Briefing Paper No. 98 January 2011

The Uses of the Afrobarometer in Policy Planning, Program Design and Evaluation
Introduction This discussion explores some specific ways in which Afrobarometer data can contribute to policy-making and implementation processes. Although it cannot and should not be the only factor that determines policy outcomes, even in a democracy, information on what the public wants has often been missing from these processes altogether, when it should be at their center. Diamond and Morlino (2005) identify responsiveness – a government that does what the people want – as the “essential result” of democracy. Yet it tends to be one of the aspects of democracy in which Africa’s emerging democratic regimes remain most deficient (Logan and Mattes 2010). And of course, it is not just governments that should care about what the public thinks. Development practitioners both within and outside of government have to understand popular priorities if they are to design effective interventions, civic educators need to know what people do – and do not – understand about what it means to be a democracy, policy and pro-democracy advocates can use knowledge of public preferences to advance their causes, and political party leaders need to measure the public mood if they are to be effective in their strategies to win popular support. There are different ways to get the measure of what it is the people want, including focus group discussions, public consultative or “town hall” meetings, formal or informal surveys, and by just talking to the “man on the street.” But all of these take effort and resources if we are to end up with really good information on popular preferences and priorities. So, too often, even when political leaders or others in the policy-making arena want to govern in ways that meet public needs, they have to rely on estimation, speculation, and secondary sources in the place of real information on popular priorities. The public attitude data provided by the Afrobarometer offers one solution to this challenge. Afrobarometer data offers a reliable – and readily available – means for taking the public’s measure, and can serve as the basis for designing policies and programs that are responsive to the priorities of African citizens. In addition, Afrobarometer data can also be utilized on the output side, to track the effectiveness of policy actors’ efforts to bring about improvements in democratic quality and governance, economic management, and social service delivery. Afrobarometer data can play a role in each of three successive steps in the policy making process:

Copyright Afrobarometer

1

1) Agenda setting and policy planning; 2) Program design; 3) Monitoring and evaluation. Agenda Setting and Policy Planning Afrobarometer data can be utilized in several different ways to contribute to agenda setting and policy planning processes. Two operate via direct indicators: respondents’ self-identified policy priorities, and their revealed policy preferences. And two are indirect: analyses of causal chains and linkages, and identification of gaps. I will discuss each of these in turn. Most Important Problems One way to learn what peoples’ policy priorities are is simply to ask them. Afrobarometer asks respondents: “In your opinion, what are the most important problems facing this country that the government should address?” Respondents can give up to three answers. Their responses provide some broad, first-cut guidance as to where governments, development practitioners, political party leaders or policy advocates should focus their energies. We generally find that across all countries, economic management issues tend to rate among the top priorities, especially jobs and poverty. But as shown by the sampling of responses in Table 1, not everyone’s needs are the same. For example, Malawi and Namibia are opposite each other with respect to the importance they place on unemployment (3 percent and 21 percent, respectively) and on food shortage and famine (21 percent and 3 percent, respectively). Infrastructure is a top priority in Liberia while water supply and electricity are far lower priorities, whereas electricity is a top concern in Nigeria, and water supply dominates in Burkina Faso. Burkinabes’ top priority is health care, but this issue falls well behind other concerns in Namibia, Nigeria and South Africa. Understanding which issues most concern the public is one key source of information that can serve as an initial guide for prioritizing sectors for policy interventions. Table 1: Selection of Popular Policy Priorities, 2008-2009
Problem Unemployment Poverty/Destitution Food shortage/famine Infrastructure Health and Sickness Water Supply Electricity Crime and Security Corruption Burkina Faso 6 10 13 5 18 16 1 3 2 Liberia 11 5 10 18 10 2 2 2 1 Malawi 3 9 21 7 8 12 1 3 2 Namibia 21 11 3 3 4 5 4 4 4 Nigeria 16 12 9 7 5 4 12 2 6 South Africa 20 8 3 4 5 4 3 11 5 Tanzania 1 6 5 14 14 14 5 2 4

*Respondents could give up to three answers. Percentages reflect the proportion of all substantive responses, i.e.,
other than “don’t know” or “no further response.”

Copyright Afrobarometer

2

Policy Positions While the above data offers broad guidance for agenda setting, the Afrobarometer also frequently asks respondents directly about more specific policy positions on a host of issues. These range from preferences with regard to democratic institutions and practices (Should we or shouldn’t we have multiple political parties? What about term limits?) to much more specific questions on, e.g., education or health policy (Should we have free primary education, or pay school fees but get better quality services? Does the country need to put more resources into AIDs treatment?), or the role of traditional leaders (Should they receive government salaries? Be required to remain neutral in the midst of partisan politics?). All of these questions concerning policy preferences have been included in all countries in the most recent round of the Afrobarometer. But another common means by which we include policy-relevant questions in the Afrobarometer is through the inclusion of “country-specific questions” in each country. The topics for these questions are identified by Afrobarometer National Partners in each country based on current events and popular topics of public debate. Some of the varied policy issues that were explored in Round 4 included: • Should there be public funding of political parties in Botswana? • Should there be direct election of Municipal and District Chief Executives in Ghana, or Mayors in Liberia? • Should the mixed electoral system in Lesotho be maintained or dropped? • Should MPs have minimum education requirements, or maximum age limits, in Malawi and Zambia? • Should Mozambique’s government only engage with other democracies in trade and international relations, while avoiding authoritarian regimes? • Should human rights violators in Liberia or Uganda receive amnesty or face punishment? • Should the media in Lesotho be completely free, or be subject to government regulation? • Should there be an old age pension scheme or cash transfers to the poor in Malawi? • Should government land reform efforts focus on purchasing commercial land for redistribution in Namibia? • Should all government contracts be reviewed by Parliament in Tanzania? • Should the Ugandan government give funding and top priority to resettling northern IDPs, or focus on other national needs? • Should schools in Mali teach in local languages? Should primary school be free and compulsory? • Should the government invest more in public transport in Mozambique? Obviously, it is possible to tackle an extremely wide range of policy issues with a tool like the Afrobarometer, offering an opportunity for critical public input into current policy debates. To date, however, this tool has been under-utilized. Afrobarometer aims to tackle this shortfall in Round 5 by increasingly working with key stakeholders in each country prior to conducting a survey, to solicit input into the country-specific portion of the questionnaire. This will allow the
Copyright Afrobarometer

3

policy community to take fuller advantage of this opportunity to explore relevant policy questions on the “hot topics” of the day in each country. Causal Chains and Linkages Another more indirect way of identifying policy priorities using Afrobarometer data is through analysis of causal relationships and linkages. Specifically, if a policy actor has identified a broad policy objective, we may be able to identify specific policies that can help to achieve this objective by using correlation or regression analysis and similar techniques. For example, democracy advocates are always interested in finding ways to strengthen support for and satisfaction with democracy. Afrobarometer analyses, meanwhile, have consistently shown that levels of corruption sharply reduce levels of satisfaction with democracy. This suggests that a policy agenda focused on reducing corruption would be one way to achieve the broader goal of strengthening democracy. As another example, opposition parties facing extremely low levels of support in many countries may be interested in understanding what drives these negative evaluations, so that they can undertake efforts to boost their image. Regression analysis reveals that women tend to have far more negative attitudes toward the opposition than men, while the poor, on average, offer more positive assessments than the wealthy. This information can serve as the starting point for developing policies or programs that might help opposition parties to tackle their image problems. Note, however, that this approach may produce a different policy agenda from that derived purely from focusing on the public’s priorities as discussed above. For example, across 20 countries, only 3 percent of respondents identified corruption as a “most important problem,” placing it well down on the list (and the highest level of concern registered in any country was just 6 percent). Thus, Afrobarometer does not always settle the question of what the “right” policy objective is – policy actors must bring their own goals and objectives to the table as well. But the Afrobarometer can provide concrete guidance to assist policy actors in achieving their objectives through access to better information about the popular will. Find the Gap Finally, one additional way in which the data can be used to build a policy agenda is by, as above, identifying a broad objective, and then using Afrobarometer data to “find the gaps.” As an example, consider again the broad objective of strengthening democracy. Afrobarometer has recently developed measures of various qualities of democracy. A sampling of these is shown in Table 2. Perusing these data, we observe that most Africans are relatively satisfied with the level of freedom they now enjoy in their societies (although Nigeria and especially Zimbabwe are notable exceptions). The level of electoral competitiveness is also rated quite highly across most countries (though Nigeria and Zimbabwe are again the exceptions, along with Kenya and Uganda). But we can also identify gaps – i.e., elements of a democratic system that are still quite weak – both at the country and the regional level. For example, representation and responsiveness remain weak, and this is true in essentially all of the countries in the Afrobarometer. Many report that their local government councilors and MPs only rarely listen to them, and MPs are typically reported to be infrequent visitors to their constituencies. Thus, as
Copyright Afrobarometer

4

Mike Bratton has observed, “this key result suggests that, to make a reality of democratic governance, policy actors ought to pay closer attention to policy measures that address the representation gap” (see Afrobarometer Briefing Paper No. 93). Table 2: A Selection of Quality of Democracy Indicators, 2008-2009
Country Botswana Ghana Madagascar Burkina Faso Kenya Zimbabwe Nigeria 20-country Mean
Source: Logan and Mattes 2010.

Freedom 3.6 3.3 2.6 2.6 3.0 1.9 2.3 2.9

Competition 3.4 2.9 2.7 2.7 1.6 1.6 1.6 2.4

Vertical Accountability 2.6 2.7 2.0 1.9 2.0 1.8 1.4 2.0

Responsiveness 2.0 1.9 1.7 2.0 1.5 1.7 1.3 1.6

We can think of numerous other examples of a “find the gap” approach to policy making. . For example, if we adopt the broad objective of strengthening the Rule of Law in Botswana, we can again use Afrobarometer data to determine where the strengths and weaknesses currently lie in the legal and security systems in Botswana. By looking at such factors as the level of trust in legal and security institutions, equality of access to legal protections, the effectiveness, equality and honesty of law enforcement, and the experience of petty bribery, we can identify gaps, and then design a policy agenda aimed at filling those gaps. In the case of Botswana, the country gets generally high marks relative to all other countries in terms of the rule of law. We find that while trust in the courts is quite high (72 percent trust somewhat or a lot), there is nonetheless a perception that more than four out of ten judges and magistrates (42 percent) are involved in corruption to at least some degree. There is also an enforcement gap of about 12 percent (the difference between those who think it highly likely that government officials will be punished for committing a crime, and those who think it likely that ordinary individuals would face punishment in similar circumstances). Thus, there remains room for improvement even in Botswana, and this “gap analysis” helps identify the particular areas of shortfall, and hence a policy agenda for the sector. Program Design A similar “find the gaps” approach is also one of the most useful tools the Afrobarometer has to offer at the program design stage as well. In this case, once a policy objective has been identified, we can use gap analysis to identify programming objectives. For example, if the Ministry of Health in Mozambique has settled on improving access to basic health care services as its policy objective, we can use Afrobarometer data to identify the regions or populations that are most underserved. We can quickly determine, for example, that while 85 percent of residents in Tete Province enjoy ready access to health clinics, only about 10 percent of those in Inhambane and Maputo Provinces (not Maputo City), and 20 percent in Gaza can say the same.
Copyright Afrobarometer

5

Alternatively, the Ministry might decide instead to focus on increasing satisfaction with existing services rather than on increasing access. Afrobarometer data reveal that despite the limited access, 75 percent in Gaza and Inhambane are relatively satisfied with the government’s efforts, while in Maputo Province only 33 percent are satisfied with the status quo, suggesting that Maputo Province might be the best target for a quality improvement program. Another key way in which the Afrobarometer can contribute to program design is through its potential value as an in-depth assessment tool. For example, the civil servants in the Ministry of Health may now know that the population of Maputo Province is quite unhappy with their performance. But then the next question in the program design process is why? Are there shortages of doctors or medicines? Do services cost too much? Or are they treated poorly by clinic staff? In-depth data on health services collected in Round 3 (2005) in Mozambique indicates that waiting times, absent doctors, and shortages of medicines are the most common problems (Table 3). Note that Afrobarometer surveys do not always carry the necessary in-depth questions on any given policy or program area to offer this kind of detailed assessment. But again, by working together with an Afrobarometer National Partner, it may be possible for a policy actor to insert a battery of questions that can provide this kind of more detailed assessment into a survey. Table 3: Problems with Public Clinics, Maputo Province, Mozambique, 2005 (percent) Never    
 
Long waiting times Absent doctors Lack of medicines or supplies Lack of attention/respect Dirty facilities Too expensive Demands for illegal payments 2 9 20 25 25 39 42 Once or twice 8 20 16 19 2 19 5 Few times 13 23 28 17 28 14 17 Often 66 27 23 25 27 16 16 No experience or Don’t know 13 20 13 14 19 13 20

“Have you encountered any of these problems with your local public clinic or hospital during the past 12 months?”

Monitoring and Evaluation Finally, we turn to the question of how Afrobarometer data can be utilized for monitoring and evaluation. Most of us face the challenge of finding ways to measure our progress, whether we are answering to our bosses, our funders, our partners, our constituents, or ourselves. Most of us recognize the need to have good measures that tell us whether or not we are achieving our targets. But we are also intimately familiar with the challenges of identifying the appropriate indicators, and of gathering the data required to actually track them. Afrobarometer data, again, may offer some solutions.

Copyright Afrobarometer

6

Whether it is for an civic education program that aims to increase women’s political participation, a local government strengthening program that targets the responsiveness and effectiveness of local government councils, or an infrastructure program that aims to increase satisfaction with government’s maintenance of urban roads and bridges, the Afrobarometer has indicators that may be relevant for program monitoring and evaluation. In fact, as a glance through a single Afrobarometer Summary of Results reveals, the Afrobarometer offers a host of indicators both in the Democracy/Governance sector, as well as across social service delivery, poverty and food security, and many other sectors. All of these indicators can be disaggregated by country, region, gender, age, education level, ethnic group, and by numerous other factors. Thus, the Afrobarometer has potential as an extremely rich source of high quality measures for monitoring trends and tracking change in key sectors in Africa (Table 4). Table 4: Selection of Afrobarometer Indicators for Liberia, Disaggregated by Gender and Urban-Rural Location
Urban Rural Male Female Total Here is a list of actions that people sometimes take as citizens. For each of these, please tell me whether you, personally, have done any of these things during the past year. If not, would you do this if you had the chance: Got together with others to raise an issue? Would never do this Would if had the chance Once or twice Several times Often Don't know 21 23 20 25 10 1 13 16 23 36 11 1 14 14 21 37 14 1 20 25 23 24 8 1 17 19 22 31 11 1

How well or badly do you think your local government is practicing the following procedures: Consulting others (including traditional, civic and community leaders) before making decisions? Very Badly Fairly Badly Fairly Well Very Well Don't know 32 36 23 7 3 33 28 29 6 4 33 29 26 9 2 31 34 26 4 5 32 32 26 6 4

Now let’s speak about the performance of the present government of this country. How well or badly would you say the current government is handling the following matters, or haven’t you heard enough to say: Maintaining roads and bridges? Very Badly Fairly Badly Fairly Well Very Well Don't know 21 28 39 13 0 47 22 21 9 1 35 25 29 10 0 34 24 30 11 1 35 25 30 11 1

It is, however, important to keep in mind that there are some limitations on our ability to use Afrobarometer data in this way, which are largely related to the size of a normal Afrobarometer sample relative to the typical scope of program implementation and impact. It all boils down to the margin of error associated with any given indicator, and consequently the question of how
Copyright Afrobarometer

7

much change is “enough” to be sure that it is “real” – i.e., statistically significant. Any result generated by the Afrobarometer comes with a margin of error based on the sample size, as shown in Table 5. If we are comparing the results of two polls with the intention of showing that there is a statistically significant difference between them, then in general the margin between the two must be at least 1.4 times the margin of error.1 At the country level, the margin of error for Afrobarometer polls is usually just +/-3 percent (2 percent in Nigeria, South Africa and Uganda, where sample sizes are 2400). So, for example, if we compare perceptions of corruption in Senegal in 2005 and 2008, the difference must be at least 4.2 points to achieve statistical significance. From Figure 1 we can see that Benin (-16 points), Namibia (-6 points) and Madagascar (-5 points) all had statistically significant declines in corruption in the Office of the President between 2005 and 2008, while Malawi (-2 points) did not. Similarly, the 8-point increase in Senegal and the 17-point increase in Mali clearly exceed significance requirements, but the 4-point change in Mozambique and the 3-point increase in Kenya do not. Of course, if we want to make comparisons across sub-national units or groups (e.g., Ghanaian men compared to Ghanaian women, sub-national comparisons across regions or provinces, or across rural-urban divides) where the sample sizes are smaller, then the difference between two numbers must be greater to reach the threshold of significance. Table 5: Margin of Error (95% confidence level) N    
 
2000 1000 800 600 400 200 100 Margin of error +/-2.2% +/-3.1% +/-3.5% +/-4.0% +/-4.9% +/-6.9% +/-9.8%

Thus, a key question in determining whether Afrobarometer data can serve as a suitable tool for program monitoring and evaluation is how much impact on a given indicator can be reasonably expected, over how large an area or population, and over what period of time. If, for example, we are implementing a national-level anti-corruption program in Kenya, and we expect it to have substantial (at least 5-10 point), national-level impacts on public perceptions of corruption over a period of 3 to 5 years, then the Afrobarometer’s current corruption measures could serve quite well as indicators for monitoring and evaluation of the project.

1

Note that this applies if the sample size of the two polls have the same sample size. The formula must be adjusted slightly if the polls have different sample sizes. See Franklin (2007) for detailed explanations and formulas.
Copyright Afrobarometer

8

Figure 1: Changes in Perceived Corruption, Office of the President, 2005-2008 (percent “most of them” or “all of them”)
50 45 40 35 30 25 20 15 10 5 0

43 26 29 25

42 27 22 16 19 17 19 11 6 10 14

27

2005

2008

“How many of the following people do you think are involved in corruption, or haven’t you heard enough about them to say: The President and Officials in his Office?” (percent answering “’most of them” or “all of them”

If, on the other hand, we are implementing a local government strengthening program in just 12 of Zambia’s 72 Districts, representing about 15 percent of the national population (and therefore 15 percent of a typical Afrobarometer sample, or 180 respondents), then the situation is a bit different. The margin of error for a sample with N=180 is +/- 7 percent. Thus: • If we are want to compare the attitudes or behaviors of the population in these 12 districts in 2008 and 2011, we would need to see at least a 10-point difference2 between the two measurements to reliably demonstrate real change. • Alternatively, if we want to establish whether there is a statistically significant difference between these 12 districts (n=180, margin of error of +/-7 percent) and the remaining 60 (n=1020, margin of error=+/-3 percent), our indicators for the two areas would need to differ by 8 points in order to demonstrate statistical significance.3 So determining whether an Afrobarometer measure can serve as a suitable indicator for program monitoring and evaluation purposes depends in large part on whether or not achieving differences of the required magnitude is realistically achievable. One way to overcome these problems, however, is to increase the size of the Afrobarometer sample, and in particular, to increase the N for the target populations. There are three possible approaches:
2 3

1.41 * 7 = 9.8 When sample sizes are different, the standard error of the difference is the square root of the sum of the squares of the standard error from each of the individual samples.
Copyright Afrobarometer

9

1) By increasing the national sample size (e.g., from 1200 to 2400 respondents), we can increase the opportunities for analysis and evaluation for many sub-groups; 2) By adding a random oversample among the target population, e.g., in the 12 districts included in the Zambian local government program we can increase N, decrease the margin of error, and thus reduce the amount of change that has to be observed to ensure statistical significance; 3) By conducting a separate survey in a targeted area or among a targeted population – In cases where target populations are small or scattered and random sampling will not capture enough members of the target population it may be most useful to conduct an “Afrobarometer-style” surveys – utilizing the Afrobarometer methodology, instrument, and partners, and thus producing findings comparable to Afrobarometer results – but including only members of the target population. For example, to test the effects on political attitudes among 10,000 participants in a civic education program will require a targeted sample within that population, but the results could them be compared to the national-level results produced by the Afrobarometer. One example of a successful utilization of the oversample approach comes from Uganda, where two new projects focused on strengthening local governments and multiparty democracy are being implemented in 16 of the country’s 80 districts. During the Round 4 survey in 2008, these two projects sponsored an oversample of 100 cases per district (total oversample N=1600), that were conducted simultaneously with the regular Afrobarometer Uganda Round 4 survey. Some additional items were also added to the questionnaire for assessment purposes and to provide program-specific indicators for the project monitoring and evaluation. Ultimately, the 2008 survey will serve as the baseline, with the intention to conduct similar follow-up surveys during and/or after project implementation in order to measure the project’s impact on these indicators. Afrobarometer data from the other 62 districts not included in these programs will serve as the control. This collaboration should offer the opportunity for project managers in Uganda to demonstrate impact most easily at the collective (i.e., 16 districts combined) level, but possibly also at the level of individual districts. Ultimately, the best approach to monitoring and evaluating the impacts of a specific policy or program, and in particular the suitability of Afrobarometer data for this purpose, must be determined on a case-by-case basis. But we believe that the full value of Afrobarometer data as a tool for monitoring and evaluation remains untapped, and we will seek to work more closely with stakeholders in Round 5 in order to make fuller use of the data for these purposes. Conclusion As demonstrated, Afrobarometer data can be utilized in many ways within policy planning, program design, and monitoring and evaluation processes. But the data is still underutilized for these purposes. Steps that can be taken to better tap this resource within the community of policy actors in Africa include, from the Afrobarometer’s side: • Soliciting more input from stakeholders in the policy community during the process of questionnaire design (especially in design of country-specific questions);

Copyright Afrobarometer

10

• •

Publicizing the opportunities and methods for utilizing Afrobarometer data for assessments and evaluations; Continuing to produce analyses that explore linkages and causal relationships on policyrelevant topics.

From the side of policy actors, you can: • Assist Afrobarometer partners to identify key policy-relevant topics to be explored in the questionnaire, especially in designing the country-specific questions for each country; • Assess whether Afrobarometer indicators are relevant to your programs and activities; • Consult with Afrobarometer National Partners about opportunities to analyze existing data or ways to expand the Afrobarometer data resource in ways that can contribute to achieving your policy making, programming and monitoring and evaluation needs. References Diamond, Larry, and Leonardo Morlino (eds.). 2005. Assessing the Quality of Democracy. Baltimore: Johns Hopkins University Press. Franklin, Charles H. 2007. “The ‘Margin of Error’ for Differences in Polls.” Available at http://abcnews.go.com/files/PollingUnit/MOEFranklin.pdf. Logan, Carolyn and Robert Mattes. 2010. “Democratizing the Measurement of Democratic Quality: Public Attitude Data and the Evaluation of African Political Regimes.” Paper presented at the Annual Meeting of the International Studies Association, New Orleans, LA, 19 February.

This Briefing Paper was prepared by Carolyn Logan, Deputy Director of the Afrobarometer. The Afrobarometer is produced collaboratively by social scientists from 20 African countries. Coordination is provided by the Center for Democratic Development (CDD-Ghana), the Institute for Democracy in South Africa (Idasa), and the Institute for Empirical Research in Political Economy (IREEP) in Benin. We gratefully acknowledge the generous support of the Canadian International Development Agency (CIDA), the UK Department for International Development (DfID), the Royal Danish Ministry of Foreign Affairs (RDMFA/DANIDA), the Swedish International Development Agency (SIDA), and the United States Agency for International Development (USAID) for Afrobarometer Round 4 research, capacity building and outreach activities. For more information, see: www.afrobarometer.org    
 

Copyright Afrobarometer

11