CRRC-GEORGIA’S RESEARCH GUIDELINES
1. INTRODUCTION
1.1. This document provides a brief overview of the research methods and professional standards that CRRC-Georgia follows. The aim of the document is to inform CRRC-Georgia researchers, donors, contractors, partners and all other interested parties about what CRRC-Georgia does and how it does it. CRRC-Georgia guarantees that all its employees and contractors comply with the standards described in this document.
2. ABOUT CRRC – GEORGIA
2.1. CRRC-Georgia is a non-govermental, non-profit research center that gathers, analyses and publishes empiricial data on the ongoing social, economic and political processes in the country.
2.2. CRRC-Georgia provides all interested parties with research results, databases, professional trainings and specific literature.
2.3. CRRC-Georgia, together with CRRC-Armenia and CRRC-Azerbaijan, constitutes a network with common goals and principles.
2.4. The goals of CRRC-Georgia are:
- Analyzing ongoing processes in society based on empirical evidence and publishing the results of research;
- Promoting informed decision making in governmental and non-governmental organizations;
- Establishing professional standards in public research.
2.5. CRRC-Georgia is governed by its Board of Trustees which is composed of Georgian and foreign members.
2.6. The research activities of CRRC-Georgia are supervised by the CRRC International Academic Advisory Board that is composed of local and foreign experts knowledgeable about the region.
3. GENERAL PRINCIPLES
3.1. CRRC-Georgia conducts research only if it can ensure high quality implementation and make the results public within a reasonable timeframe.
3.2. CRRC-Georgia strictly adheres to ethical research standards (e.g.: AAPOR).
3.3. CRRC-Georgia believes that debates and constructive criticism about research methods and findings are integral parts of public research.
3.4. The research activities of CRRC-Georgia are nonpartisan and neither activities nor the organization aim to support or oppose any political party or politician.
4. QUANTITATIVE RESEARCH
4.1. Types of quantitative research
4.1.1. Depending on research goals, CRRC-Georgia uses several types of quantitative research:
One-time survey – this type of survey provides information on the research topic during the survey period.
Longitudinal survey – the same survey is repeated at certain intervals, but respondents are sampled for each “wave” independently. The aim of this kind of survey is to determine social, political, economic and other trends in the country.
Panel survey – the same survey is repeated at certain intervals and the same respondents are interviewed for each “wave.” The aim of such a survey is to track changes at the individual level.
Experimental survey – control and experimental groups are identified during such a survey and the results are compared between these two groups.
4.1.2. Quantitative survey interviews can be conducted face to face or be self-administered, and are completed on either the telephone or the internet:
- During face to face interviews, the interviewer usually fills out the questionnaire at the respondent’s dwelling. Respondents do not have access to the questionnaire during the interview process. Face to face interviews are conducted using printed questionnaires (PAPI) or computer based questionnaires (CAPI).
During computer based surveys the interviewer enters respondents’ answers in a special program installed on tablet computers. At the end of the interview, the results are sent to our server via mobile internet connection. Compared to the surveys with printed questionnaires, the advantages of such surveys include faster fieldwork and lower expenses. - During computer based, face to face surveys CRRC-Georgia uses tablet computers, which are based on the Android operating system and the open source software – ODK Collect. The time and duration of an interview is automatically recorded in the computer and skips in the questionnaire are programmed in advance. Also, it is possible to obtain integrated geographical information through GPS built into the tablet computers.
- During self-administered surveys, respondents fill out questionnaires themselves in the presence or absence of an interviewer (e.g.: via internet). As with face to face interviews, self-administered surveys can be conducted using printed questionnaires (PAPI) or computer based questionnaires (CAPI).
During online surveys, the opinions of specific social groups are studied. Respondents fill out special forms that are made using a number of computer programs (Google forms, Surveymonkey, Limesurvey etc.). - Computer based telephone survey (CATI) is the fastest method of gathering information. It is administered from the CRRC-Georgia office. Interviewers record respondents’ answers in a computer program written specifically for this purpose.
4.2. Questionnaire
4.2.1. While working on the questionnaire, CRRC-Georgia ensures that questions asked in the survey will gather reliable information about the research topic. In the process of questionnaire development, CRRC-Georgia closely cooperates with interested parties.
4.2.2. In its questionnaires CRRC-Georgia uses questions from international surveys and also develops new, original questions.
4.2.3. CRRC-Georgia pays special attention to the formulation of questions as well as their sequence.
4.2.4. All questions have corresponding instructions (e.g.: number of answers, should an interviewer read out an answer or not, etc.).
4.2.5. If necessary, questions are accompanied with cards which are shown to respondents. Response options are listed on the card.
4.2.6. In order to ensure high quality data, it is recommended that the duration of face to face interviews not exceed 50 minutes and the duration of telephone interviews – 15 minutes. Therefore, the number of questions is limited.
4.2.7. Prior to fieldwork CRRC-Georgia conducts a pilot survey in order to reveal and correct mistakes and inconsistencies in the questionnaire. Generally, 30 to 50 interviews are done during the pilot survey.
4.3. Sampling
4.3.1. CRRC-Georgia utilizes a probability sampling approach in most of its quantitative research. However, depending on the research objectives, non-probability sampling approaches can sometimes be used, e.g. “snowball” method.
4.3.2. For most surveys, CRRC-Georgia uses lists of electoral precincts from the Central Electoral Commission as a sampling frame. The list includes a number of registered voters per electoral precinct and the precinct boundaries. The occupied territories of Georgia, special electoral precincts and inaccessible settlements are excluded from the sampling frame.
4.3.3. CRRC-Georgia uses stratified multistage clustered sampling for most surveys. The target population is divided across a number of strata – e.g. urban and rural populations or employed and unemployed populations, etc. The sample size is calculated for each stratum with consideration to the expected non-response rate. For example, if 1000 completed interviews are needed for the survey and the expected response rate is 80 percent, the sample size (number of attempted contacts) will be 1250.
4.3.4. Primary Sampling Units are the electoral precincts that are sampled within each stratum randomly, with probability proportional to size. The measure of size of the electoral precinct is the number of registered voters within the precinct.
4.3.5. Within electoral precincts, a predefined number of households are sampled based on a probability sampling approach, using the random walk method. Interviewers are given the number of households to sample within the electoral precinct and the address of the starting point, along with relevant instructions.
4.3.6. One respondent within each household is also selected based on a probability sampling approach using the Method for Respondent Objective Selection developed by Leslie Kish, the “Last birthday method,” etc.
4.3.7. The described sampling design allows inference of survey results on the total population, as well as on each stratum.
4.4. Fieldwork
4.4.1. CRRC-Georgia has its own supervisors in all of the regions of the country, with the exception of the occupied territories. Supervisors manage their network of interviewers and control fieldwork in their regions.
4.4.2. Prior to fieldwork, supervisors and interviewers go through intensive training, wherein they become familiar with household and respondent selection procedure, as well as receive instructions on the interview process.
4.4.3. Fieldwork is controlled at two levels – by supervisors and CRRC-Georgia employees who are responsible for the fieldwork.
4.4.4. Supervisors check and submit completed questionnaires at several stages. In the case of mistakes or inconsistencies, interviewers undergo additional training and conduct interviews again.
4.4.5. Supervisors fill out fieldwork report forms where they describe the fieldwork process and specify any events that might have effected respondents’ answers.
4.4.6. For quality assurance purposes, after the completion of fieldwork, CRRC-Georgia back checks 5-10 percent of the completed interviews. If violations of survey standards are found, interviews are canceled and new interviews are conducted.
4.5. Databases
4.5.1. CRRC-Georgia employs the computer program CSPro for entering PAPI questionnaires. The software ensures data quality control and allows export into formats compatible with different statistical programs.
4.5.2. Data entry operators are experienced and specially trained individuals whose work is overseen by the data entry supervisor.
4.5.3. The process of creating a database from PAPI questionnaires is as follows:
Preparation of data entry form – first, the data dictionary is prepared where the particularities of the variables are defined: name, length, type, possible values, etc. The next step is creating the data entry form based on the dictionary. The form programmatically defines skip patterns and visually resembles questionnaire design.
Testing the data entry form – at this stage compliance between form and questionnaire is checked, and discrepancies are unveiled and corrected.
Operator training – after the form is prepared and tested, data entry operators attend special trainings on how to use the program. Namely, they are instructed which questions need the most attention and in what cases they should approach the supervisor. Additionally, the operators enter several questionnaires for testing and training purposes.
Double data entry – in order to ensure the quality of the database, each questionnaire is entered twice by two separate operators (double blind entry). The entry process is conducted in a specially arranged computer lab. The computers are connected to a network and data is collected on a server where the data entry supervisor can perform quality control and assessment of the process.
Data comparison and preparation of raw database – after the entry process, the data entry supervisor performs a comparison of blind data entry results. All discrepancies are checked and are corrected in the database. The process ensures the full compliance of the dataset with the filled in questionnaires.
4.5.4. When CAPI method is used, the form which is based on XML standard is prepared using Microsoft Excel and then converted into the corresponding file format. The completed form is uploaded to a dedicated server (Google App Engine) which is used for data and form storage.
4.5.5. Only a small circle of researchers has access to the questionnaires for both methods of data collection. The anonymity of respondents is fully protected.
4.6. Weighting
4.6.1. In order to make inferences from respondents’ answers about the target population, the survey data is weighted. Weights are composed of three components:
Probability weights – as members of the target population have a different probability of being sampled, each sampled respondent represents the views of a different number of persons. Therefore, each respondent is given a weight coefficient that is the inverse of his/her probability of being sampled in the survey.
Non-response weights – weight coefficients are corrected considering the response rate, e.g. if the survey response rate is 80 percent, the weight coefficient of each respondent is multiplied by 1.25.
Demographic weights − weight coefficients are once more corrected based on the population’s demographic data, in accordance to the proportion of respondents corresponding to the demographic groups in the population.
4.7. Content Analysis
4.7.1. CRRC-Georgia uses content analysis in media monitoring.
4.7.2. The goal of media monitoring is to study the coverage of certain issues and also verify how the coverage complies with journalistic standards. Media monitoring can be both quantitative and qualitative.
4.7.3. Based on research objectives, CRRC-Georgia chooses media segments (TV, printed media, radio and online media), specific media outlets, items (news, articles, talk shows, advertisement and others) and the units of observation.
4.7.4. During media monitoring, CRRC-Georgia counts the time allocated to each unit, defines the tone of coverage (positive, negative or neutral) and evaluates the compliance of coverage with journalistic standards and legislation.
4.8. Quantitative analysis
4.8.1. CRRC-Georgia analyses quantitative data using programs like SPSS, Stata, and R among others.
4.8.2. The first step in quantitative data analysis is the study of frequencies. This gives information about the distribution of respondents’ answers.
4.8.3. The second step in quantitative data analysis is cross tabulation by means of which we can study differences between demographic or other groups e.g. how the usage of the internet varies in different settlements (in urban and rural settlements). Such analysis can be very wide and diverse.
4.8.4. Scales are often devised during quantitative analysis. For example, by means of several variables, trust towards political institutions can be measured and then by using these variables, an index of trust can be calculated.
4.8.5. For academic projects, CRRC-Georgia uses more advanced techniques of data analysis, such as regression, cluster analysis, factor analysis among other forms of analysis.
5. QUALITATIVE RESEARCH
5.1. In-depth interviews and focus groups
5.1.1. CRRC-Georgia conducts in-depth interviews and focus groups when there is a need for more comprehensive information about the research topic. These methods are also often used as a first stage in the development of questionnaires for quantitative surveys or to fill in existing gaps after quantitative surveys.
5.1.2. During in-depth interviews and focus groups, respondents have the opportunity to expand upon and demonstrate their views. Therefore this method is widely used while studying controversial and sensitive issues. Data obtained through in-depth interviews and focus groups cannot be generalized.
5.1.3. In-depth interviews and focus groups are conducted using a flexible conversation guide. Their length is approximately 90 minutes.
The optimal number of focus group participants is 8-10 people. CRRC-Georgia selects participants based on research objectives. This method is very effective for studying different views around the research topic. Focus group members can also be grouped based on gender, education, profession as well as other criteria.
5.1.4. In-depth interviews and focus groups are recorded on video or audio files with the consent of respondents and afterwards are transcribed. In addition, respondents fill out demographic forms and interviewers/moderators record information about their observations.
5.1.5. CRRC-Georgia guarantees the confidentiality of participants of qualitative research.
Research participant statements cannot be linked back to them.
5.2. Ethnographic Fieldwork
5.2.1. During ethnographic fieldwork, CRRC-Georgia researchers observe the daily lives of research subjects in their usual environment. Fieldwork continues for several months. During such study, researchers are involved in the activities of research subjects and to the extent possible share their ways of life.
5.2.2. During the observation process, researchers take fieldwork notes wherein she or he describes events that took place during the study, as well as the reactions and expectations of research subjects.
5.2.3. During ethnographic study the researcher conducts formal and informal interviews, and also obtains various forms of documentation including photos and videos that are connected to the study.
5.3. Management of Qualitative Data
5.3.1. CRRC-Georgia guarantees the anonymity of all respondents who are involved in research. All respondents are assigned pseudonyms during transcription. Also, information that can somehow be used to identify respondents is deleted (e.g.: employment place, place of study, year of birth, etc.).
5.3.2. During the preparation of transcripts CRRC-Georgia marks all details that can be important for the research. For example, this could include an ironic or cynical tone from the respondent.
5.4. Analysis of Qualitative Data
5.4.1. During qualitative analysis, CRRC-Georgia describes, explains and systemizes collected information.
5.4.2. In most cases, during the qualitative analysis, CRRC-Georgia identifies important topics and systemizes respondents’ interpretations. Some qualitative approaches like discourse analysis and narrative analysis do not always need this.
5.4.3. In order to distinguish various topics, CRRC-Georgia codes qualitative data in different thematic categories. Coding implies differentiating various parts of the text that are connected to different topics and assigning them names. This gives the researcher the opportunity to easily find and summarize all of the related parts that are dispersed throughout the text, but are connected to the same topic. Also, using this tool, researchers can observe similarities and differences between cases.
5.4.4. For coding and analyzing qualitative data, CRRC-Georgia uses the computer program QSR NVivo which is specifically designed for qualitative data analysis. This program gives researchers the ability to organize and code immense amounts of qualitative data (texts, audio and video materials, etc.)
6. DISSEMINATION OF RESEARCH RESULTS
6.1. Availability of the Data
6.1.1. CRRC-Georgia places databases in STATA and/or SPSS formats on its website: http://caucasusbarometer.org.
6.1.2. Together with databases, CRRC-Georgia also puts appropriate documentation – research methodology, questionnaires, show cards, and fieldwork reports – on the website. With the online documentation, researchers can gain access to thorough information about the research.
6.1.3. Prior to placing databases on the website, CRRC-Georgia deletes all information that can help to identify respondents.
6.2. Online Data Analysis Program (ODA)
6.2.1. CRRC-Georgia uses an online data analysis program (ODA) for broader dissemination of survey results. With the help of this program, everybody can do data analysis and visualization easily, without knowledge of any statistical program.
6.2.2. Using the online data analysis program, interested parties can analyse data from the main surveys conducted by CRRC-Georgia. For example results of the “Caucasus Barometer” can be analysed both in historical and regional perspectives.
6.2.3. In the online data analysis program, the text of the questions and answer options are also available, and in some cases an explanation of how answer options were grouped.