Quantitative and Qualitative Impact Assessment
Quantitative Data Collection
We are conducting a longitudinal household survey, consisting of three rounds of
data collection:
- Baseline Household Survey Round 1 (March 2007) English Chichewa
- Household Survey Round 2 (September 2007)
- Final Household Survey Round 3 (March 2008)
Study Sample
The study will follow
a panel of 408 intervention and 408 control households in cash transfer
recipient and non-recipient villages.
The Mchinji District Cash Transfer Scheme Secretariat first identified intervention and control villages that have not yet received transfers. Then, they implemented the
pilot's standard village-level targeting process in both
intervention and control villages in February 2007, identifying eligible households in both types of villages.
The targeting process -- which is part of the intervention and completely separate from the evaluation -- entails the
following: The District Secretariat trains a Community Social
Protection Committee (CSPC) to help implement the scheme. The CSPC
makes a list of ultra poor labor constrained households based on
community knowledge and the local Village Headman signs off on this
list. The CSPC then visits the households to fill out an application
for each household. Next, the CSPC ranks households and chooses the 10%
poorest. A community meeting occurs where the ranking is discussed.
Then Village, District, and National Committees must approve the list
of eligible households.
After this process was complete, we approached intervention and comparison households to collect baseline data.
Transfers to study households
Intervention households will receive their first payment after the
baseline survey is completed (April 2007). Control households will
receive the transfer after the final survey (approximately April 2008).
The control villages are in the same district and are similar to
intervention villages, but they have not yet been included in the pilot. It will be fully explained to control households by the
Community Social Protection Committee that they will receive the
transfer as it is rolled out through the district. These villages are
geographically separated from recipient villages in order to avoid
potential conflict or jealousy.
Sampling Frames
The two sampling frames include a roster of all cash transfer approved
or eligible households in the (1) recipient and (2) non-recipient
village groups. We will interview the approximately 100 households per
village group identified by the Community Social Protection Committees
as the poorest 10% of households. With 4 intervention village groups and 4 comparison village groups, we will survey 408 intervention
households in four recipient village groups and 408 control households
in four non-recipient village groups. We will interview the head of
household that is formally registered to receive the grant or the
deputy that they have appointed in the cash transfer application.
Respondents
The respondent will be selected by the interviewer when he/she first
visits the household by asking to speak with the person in the
household who is registered to receive the grant.
Survey
The household survey (English Version / Chichewa Version) has been adapted from existing household surveys
used with similar populations throughout Malawi, such as the Integrated
Household Survey, Demographic and Health Survey and the Multiple
Indicator Cluster Survey. The household questionnaire will document
basic health, demographic and economic conditions, including
- Household panel with demographic data including age, gender, marital status, employment status, education,
- Health, hygiene and health seeking
behaviors
- Mobility and migration
- Orphan data
- Recent deaths in household and causes of death
- Economic data including all types of income, ownership of assets and durable goods, household expenditures
- Housing type
- Food security (including the quality
and quantity of meals, food diversity, and satisfaction with meals)
- Anthropometric measurements
- Time use and adult and child care
patterns
- Emotional well-being of household head
- Receipt of social safety nets and care and support of orphans and ill household members
- Coping behaviors
The household questionnaire will be administered to the
person registered to receive the cash transfer. Round 2 and Round 3
surveys will also contain these domains, as well as additional
indicators that emerge as important to the study.
In conjunction with the administration of the household questionnaire,
weight and height measurements of children and adults will occur at
round 1, round 2 and round 3. Weight and height measurements will be
collected using portable scale and measuring boards.
Community consultation
Before piloting the survey instrument and
between rounds of the survey, we met with community leaders and members to let them know about the evaluation, how to identify study staff and what they can expect from the study.
Study Staff
Our Field Team is literate in Chichewa
and English and everyone has previous interviewing experience and high school
level education. Interviewers are directly supervised by the field supervisors, who are conducting ongoing training in several stages. Each questionnaire and the notes made
by interviewers are reviewed by field supervisors daily in order to
pick up circumstances warranting follow-up and possible referral to
social services. The PI and Co-PI
conduct weekly daily debriefing sessions during
which any serious family/individual problems noted by the interviewers are discussed.
Pilot
We piloted the survey instrument in 42 households before conducting the first full round of the household survey. Results of the
pilot survey allowed us to revise the questionnaires and to improve
interviewers' training to ensure that the first full round of the
survey proceeds smoothly and with a minimum of disruption to the
targeted households.
Data entry, storage, and linking of datasets
Data entry and storage
will be overseen by a data manager hired for the study. Data entry
clerks at the Mchinji field office will enter questionnaire responses
into an Access database. Names and study IDs will be kept in a separate
file that is password protected and only available to the study
coordinators.
Data Analysis
Data
collected for this study will be used to assess differences in
indicators of economic, health and human development between households
based on receiving the cash transfer and the impact of the cash
transfer in recipient households over time. Each of our research
questions will require a separate analysis of the data described above.
Two categories of analysis will be conducted: household-based and
child-based, according to the relevant unit of analysis for each
research objective. For example, we will assess financial expenditures
on a household level and anthropometric data on a child level.
The primary analysis is of anthropometric data among children aged 0-7,
given that the Government of Malawi is particularly interested
in ascertaining whether the Cash Transfer has a significant impact on
the health and physical development of children.
We will first compare the child and household based measures described
above through bivariate contingency tables (with Mantel Haenszel
chi-square tests for significance of differences) using indicators of
recipient status, dependency ratios, and other household or child
characteristics.
We will then use linear, logistic, and other regression analyses to
control for such factors as household size, household composition,
dependency ratio, and personal characteristics of the head of
household. For all components of the study, independent covariates will
include:
-
Receipt of cash transfer
- Household size, composition, and dependency ratio
- Household income and wealth (scale created from asset inventory; will
not be used as covariate for the economic analyses where it is the
dependent variable of interest)
- Demographic characteristics of adults (sex, age, education level
completed, marital status, employment, relation to household head,
literacy)
- Demographic characteristics of children (orphan status, current age, age
when orphaned, sex, and number, age and location of biological
siblings).
- Whether the household receives any other outside assistance including
the source, type and frequency of support
- Whether the household has a
chronically ill member or has experienced a death in the past year
(including age of and employment status or regular activities of
deceased)
- Whether the household has experience other shocks, Loss of employment
/ cash remittances or other support, crop failure, theft, damage to
housing, change in household composition
Dependent variables for the various components of the analysis will include:
- Anthropometric measurements including height and weight data for children under age 8
- Nutrition and food security including the quality and quantity of meals, food diversity, and satisfaction with meals
- Educational outcomes for all children under the age of 18 in household,
including school enrolment, attendance, whether student has dropped
out, whether family/child wants to re-enroll, length of time out of
school, and reasons for non-enrolment
- Health, hygiene and health seeking behaviors for all household
members including health status (reported by household head),
disabilities, chronic health conditions over past year (list of
diseases), and acute illnesses in past month (fever, diarrhea,
respiratory illness, etc.); reported hygiene, such as use of soap,
clean water for cooking, bathing; and health seeking behaviors and type
of treatment for care for illnesses (health care, medicines etc.)
- Expenditure data for each household including detailed percent
spent on food and non-food items (alcohol and tobacco, clothing,
housing and utilities, transport etc), per person expenditure on
education and health care, debts, savings, investments, gifts
- Time use including (if yesterday is typical day), number of hours
providing care to children, sick adults, elderly, doing work, or
household chores; and for each child under 19 years, whether child
works in home or outside of home, type of work and number of hours per
day
- Emotional well-being of household head including an assessment of
well-being and satisfaction with life, assessment of worth;
connectedness/social support; resilience/coping skills (using standard
scales) (8) Mobility and migration in household including the number of
adults/children that have entered/exited household in past year, the
reason for migration and place that the adult/child has migrated to or
from
- Coping behaviors including selling assets, stinting, begging,
prostitution, or other risky behavior and experience of violence in the
households
- Delivery of cash transfer (in follow up surveys) including the
accessibility; reliability; timeliness; and satisfaction with transfer;
perceptions of fraud/corruption and scheme transparency; and
satisfaction with targeting process
|
 |
| Village Guides in Mchinji |
|
Qualitative Impact
The purpose of the qualitative impact assessment is to gain a deeper understanding of how cash transfers impact recipient households than we can gain from a qualitative survey. In the qualitative phase of the impact assessment, conducted in October / November 2007, our primary tools were be key informant interviews and focus group discussions, which are two complementary qualitative research methods. Key informant interviews provide a more thorough, but individual perspective on sensitive topics, while focus group discussions allow the efficient collection of a diversity of opinions on specific topics. When combined, these two techniques permit the investigation of both the breadth and depth of knowledge and attitudes about an issue with a high degree of social and cultural relevance. We conducted key informant interviews with recipient households in the intervention villages and with local stakeholders.
Key Informant Interviews
During these interviews, we intend to generate more detailed data and information on sensitive topics related to the specific ways that the cash transfer has impacted the household, including the scheme’s ability to mitigate the impact of AIDS, changes in the households’ spending and care patterns, hygiene and health seeking behaviors, mobility and migration, violence in the household, child labour and other sensitive issues. We will identify households through a simple random sampling of all recipients in the intervention villages, stratified by age (<25 years, 26-49 years, 50+ years) and gender so that we obtain detailed information from a variety of households. We will tentatively interview the head of approximately 80 recipient households, though the final number of interviews will be determined during the process of data collection, as the interviewers will continue to conduct interviews until they feel they are no longer obtaining valuable or unique information.
In addition, we will conduct approximately 30 key informant interviews with local stakeholders including representatives from the following organizations: Health care facilities, Department of Social Welfare, Schools, Police department, Agricultural services, AIDS support organizations including NGOs, CBOs, FBOs, and other key stakeholders. The purpose of these interviews is to assess positive and negative perceptions regarding the influence of the cash transfer scheme; the scheme’s ability to mitigate the impact of AIDS; stakeholder perceptions of recipient behaviors regarding livelihoods, time-use, and health seeking behaviors; the community’s perceptions of the fairness of targeting; the existence of jealousy among non-beneficiary households; and perceptions of possible economic multiplier effects. Trained bi-lingual (Chichewa/English) interviewers will conduct all interviews. Each interview will take place in a private location convenient to the participant and, to limit the burden on participants, interviews will be limited to a maximum of 60 minutes. Prior to the beginning of primary data collection, the evaluation team will conduct training in field methods and ethical issues for interviewers. Interview guides will be pilot-tested before being finalized.
Focus Group Discussions
We also conducted approximately 20 focus groups gathering stakeholders and community members from intervention villages. The purpose of the focus groups is to learn about community perceptions of the impact of the cash transfer on recipient households and on the overall community. For example, we will conduct focus group discussions with members of households that were initially identified as ultra poor but were eliminated during the ranking process and labeled as not eligible for benefits. The purpose of these interviews is to assess the attitudes of these households’ towards the targeting process and determine the level of jealousy that persists among these households. We will conduct additional discussions with groups including village chiefs, women’s groups, and other key stakeholders.Trained, bi-lingual (Chichewa/English) interviewers will also facilitate the focus groups. Discussions will last approximately 1 hour and follow a semi-structured question guide to ensure that specific topics are covered, but also allow for an open-ended exchange of information among participants. The interviewer will be able to probe when unexpected, but relevant, responses emerge. Participants will also be asked to ‘free list’ regarding the impact of cash transfers grants on the community. After generating these lists, participants will be asked to prioritize their responses. Such collated responses will provide useful data about perceptions of cash transfers, with a relative sense of their perceived importance.
Qualitative Data Analysis
The primary purpose of the qualitative analysis is to identify the important themes, patterns, and concepts that we were unable to gather during the quantitative survey or to get into more depth than the survey allows. Therefore, we will conduct a content analysis of the qualitative interviews and focus groups to assess how individuals responded to given topics. Through focused attention where we read and reread the data, we will identify salient themes and patterns of ideas, concepts, and terminology related to the key topics. We will also search for deviations from the common themes and patterns and the additional factors that might explain atypical responses. We will look for interesting anecdotes that provide insight into our broader study questions as well as findings that suggest that we need to focus on additional questions to help link research questions and themes. In addition, we will assess corroborating and divergent views between study participants, revisiting the data for plausible explanations and linkages between the various salient themes and patterns. We will continue to categorize data until all relevant themes and patterns have been identified.
|
|
