MANUAL ON POOLING OF CENTRAL AND STATE SAMPLE DATA

Transcription

1 क न द र य और र ज य प रत दर श समक क एक करण क प स त क र प र स 66 व द र MANUAL ON POOLING OF CENTRAL AND STATE SAMPLE DATA NSS 66 th Round प ररव ररक उपभ क व यय HOUSEHOLD CONSUMER EXPENDITURE सम क ववध यन प रभ ग र ष ट र य प रत दर श सव क षण क य शलय र ष ट र य स स ययक य स गठन स स ययक और क यशक रम क य शन द वयन म लय भ र सरक र Data Processing Division National Sample Survey Office National Statistical Organization Ministry of Statistics & Programme Implementation Government of India

2

3

4 CONTENTS Subject Page No. Part-I: Methodological Issues List of abbreviations used in the Compendium 1 Chapter1: Introduction 2-5 Chapter2: Sample design and concepts and definitions 6-10 Chapter 3: Pooling Methodology Part-II: State Results Chapter 4: Pooled Results of Andhra Pradesh Chapter 5: Pooled Results of Bihar Chapter 6: Pooled Results of Gujarat Chapter 7: Pooled Results of Himachal Pradesh Chapter 8: Pooled Results of Karnataka Chapter 9: Pooled Results of Kerala Chapter 10: Pooled Results of Odisha Chapter 11: Summary Results AP.1-19 BHR.1-19 GUJ.1-19 HP.1-12 KTK.1-19 KRL.1-17 ODI.1-19 SUM.1-8 Part-III: Annexure Annexure-I: Estimation Procedure ANX-I.1-5 Annexure-II: District-wise allocation of FSU - Rural Annexure-III: District-wise allocation of FSU - Urban ANX-II&III.1-5 ANX-II&III.6-9 Annexure-IV: Facsimile of Schedule 1.0 Type-1 Schedule 1.0:1-21 (Type-1) Annexure-V: Facsimile of Schedule 1.0 Type-2 Schedule 1.0:1-20 (Type-2) ii

5 Part-I METHODOLOGICAL ISSUES (Sample Design, Pooling Methodology, Poolability Tests etc.)

6 List of abbreviations used in the Manual Abbreviation DES DPD FSU HH IV MPCE MR MRP MMRP MSE NSC NSSO RSE SSB UFS URP USU Description Directorate of Economics and Statistics Data Processing Division First Stage Units Household Inverse Variance Monthly Per-Capita (Consumer) Expenditure Matching Ratio Mixed Reference Period Modified Mixed Reference Period Mean Standard Error National Statistical Commission National Sample Survey Office Relative Standard Error State Statistical Bureau Urban Frame Survey Uniform Reference Period Ultimate Stage Units Manual on Pooling of Central and State Sample Data Page 1

7 Chapter1: Introduction The National Sample Survey (NSS) was set up in 1950 under Indian Statistical Institute (ISI), to bridge large gaps in statistical data needed for planning, policy formulation and computation of national income aggregates, especially in respect of the unorganized and household sector of the economy. The NSS was re-organized into National Sample Survey Office (NSSO) in 1972 under the Ministry of Statistics and Programme Implementation. Since then NSSO has been conducting nationwide multi-subject, integrated, large-scale sample surveys in the form of successive rounds covering various aspects of social, economic, demographic, industrial and agricultural statistics. These surveys are undertaken striking a balance between the urgent and contemporary need for reliable statistical data on different topics and the constraints of limited resources, both physical and financial. 1.1 Organizational structure and main activity of NSSO: The NSSO has 4 Divisions namely Survey Design & Research Division (SDRD), Field Operations Division (FOD), Data Processing Division (DPD) and Coordination & Publication Division (CPD). The SDRD is located at Kolkata and responsible for planning the Survey, formulation of sampling design, drawing up of survey schedules, formulation of concepts and definitions, preparing field instructions for canvassing schedule, preparation of scrutiny check points, drawing up tabulation programme and finally release of survey results and reports. The FOD with Headquarter located at New Delhi and Faridabad, six Zonal offices located at Bangalore, Kolkata, Guwahati, Jaipur, Lucknow and Nagpur, 49 field offices at Regional level and 118 field offices at sub-regional level spread over India is primarily accountable for data collection of NSS surveys with mechanism of rigorous data quality checking through inspection, scrutiny checks, training and managing the field operations in time bound manner. The DPD is mandated with processing of survey schedules canvassed in the various socioeconomic surveys in respect of central sample. The Division s headquarter is located at Kolkata in the same building just peer to SDRD with 8 Data processing Centers located at Kolkata, Nagpur, Delhi, Giridih, Ahmedabad and Bangalore. DPD is responsible for sample selection and preparation of sample list for central and state sample, pre-data entry scrutiny of filled-in-schedules, in-house development of software for all stages of data processing right from sample selection to final table generation, data entry and verification of filled-in schedules with double entry system, validation of data at different phases covering both content and coverage checks, data editing, preparation software for multiplier calculation and tabulation, generation of tables as per tabulation plan and finally dissemination of unit level data of central sample to public domain through Computer Centre located at New Delhi. Providing technical assistance to states and sharing data processing instruments for processing their state sample data, providing unit level data of central sample to states for the purpose of pooling have also became a regular feature of DPD. All matters of coordination between various Divisions of NSSO, preparation of action plan for NSS surveys and interaction with line ministries/departments on NSS matters, dissemination of survey results through the technical journal Sarvekshana and organising national seminars to discuss the survey results etc are done by CPD. 1.2 Need for pooling: The NSSO has maintained its standard in encompassing conceptualizing, planning, designing, organizing field operations, data processing, data validation and report generation of large scale multi-subject sample surveys over the years. NSS data from 38 th round to 69 th round are currently available for dissemination at computer Manual on Pooling of Central and State Sample Data Page 2

8 centre. NSSO generates all-india and State levels estimates and reports thereof, based on central sample, which is covered under its integrated framework. Although, the State level estimates of key parameters generated by NSSO are considered to be reliable, doubts are expressed over robustness of these estimates at further disaggregated level. However, NSSO does not generate district level estimates based on its surveys due to inadequate sample size although there has been a burning need of generating such district level parameters from NSS data for the benefit of decentralized planning. 1.3 States participation in NSS: The States started participating in the programme of collecting socio-economic data on the same subjects from the 8th round (July June 1955) of NSS using the same concepts, definitions and procedures and by adopting the same sample design based on independently drawn sample (known as state sample) as that of NSSO. Sample sizes of central and state samples are equal for most of the States/UTs (equal matching sample). But there are some States where the number of samples surveyed by State statistical agencies is double that of the size of central samples. 1.4 Pooling of central and state sample data: One of the objectives of States participation in the NSS programme is to provide a mechanism by which sample size would be increased and the pooling of the two sets of data would enable better estimates at lower sub state level, particularly at district level. At the State level, this will result in increased precision of the estimates and at disaggregated level, estimates will be more stable. But the major benefit will be derived in the case of estimates are generated at sub-state level like NSS regions/districts. Although the need for pooling central and state sample data was felt for quite some time and the 13 th Finance Commission had also made special provision for additional funds in each district to carry out this exercise, little progress was made in this respect in terms of evolving a uniform methodology of pooling and also testing for poolability of the two sets of data. While some states, of their own, pooled the results of central and state samples for a few NSS rounds, there was a complete lack of uniformity in their approach which resulted in a loss of comparability of such pooled data. It was against this backdrop that the National Statistical Commission appointed a professional committee under the chairmanship of Dr. R. Radhakrishna, Ex-Chairman, National Statistical Commission (NSC) to examine the issues. The Committee in its report gave a detailed methodology for pooling and also the tests for poolability. The committee suggested that poolability test of two sets of data must be exercised before pooling the two independent estimates derived at a particular domain using weights as inverse of estimated variances of the estimates or using weights as matching ratio of states participation in the NSS programme. Thus in contrast to earlier methodologies of pooling the data by merging two data sets and re-computation weights based on merged data the new methodology suggested the pooling of estimates using weights as stated above. Following the recommendations of the Committee, DPD took initiative to provide all kind of technical guidance and support to the states in pooling their data collected in NSS 66 th round. 1.5 Pooling exercise prerequisites and precautions: The basic requirements for pooling are that State and Central sample data should have common layout and passed through common validation checks and the data sets should be poolable. State sample data processed in other than central data layout needs to be re-casted into central layout both in terms of data content and field attributes before using the pooling software. Non sampling Manual on Pooling of Central and State Sample Data Page 3

9 error must be removed from two sets of data after following the uniform frame work both at data collection stage and data processing stage so that two sets of data are poolable. The methodology of pooling suggested in this manual is to be applied only after testing the two sets of data for poolability as suggested in the manual. The software is to be run as per the instruction taking all the necessary precautions suggested above. The results are to be interpreted cautiously based on the results of the poolability tests and also the RSE of the pooled estimates. 1.6 Layout of manual: This manual is a methodological exercise giving a comprehensive account of pooling of central and state sample NSS data in order to generate sub-state level estimates. The software developed for pooling these two sets of data is also given with this manual in a CD. CD contains the poolability test software with necessary documents for its installation and operation. It also contains separate folder with pooling software for NSS 66 th round for all the three schedules viz schedule 1.0 type-1, schedule 1.0 type-2 and schedule 10 with necessary documents such as data layout, multiplier layout, instructions, defiles and batch-files used for generating work-files from central and state sample data. Estimates for central and state sample and RSE (relative standard error) of estimates at the domain level are provided in report folder of the software in order to facilitate in building the pooled estimates from central and state sample estimates. Unit level data of two states namely Karnataka and Bihar for all the three schedules viz schedule 1.0 type-1, schedule 1.0 type-2 and schedule 10 in respect of central and state sample are also included in the CD in a separate folder after remapping the FSU(first stage unit) number so as to facilitate the users in simulating the exercise of pooling operations as per instruction mentioned in various documents of pooling software. However, one should follow the central sample data layout for using this software. The results of seven states that have already done the pooling of NSS 66 th round data using this methodology are also presented in this manual to impress upon the fact that the exercise is very much doable so as to generate sub-state level pooled estimates. Although the present exercise has been done on a selected set of socio-economic indicators based on Schedule 1.0 Type 1 and 2, the same can easily be extended for other indicators being worked out in other socio-economic household surveys. It is also very important to note here that the methodology suggested in this manual is for pooling the central and state sample estimates and not pooling the central and state sample unit level data, which is a major deviation from the earlier practices of pooling. The manual is organised as follows: Part-I of the manual gives a detailed methodological note and explains the background and need of pooling in three chapters including the present Chapter 1. Chapter 2 of part-i gives a brief account of the sample design followed in NSS 66 th round and also some important concepts and definitions. Chapter 3 of part-i gives the detailed methodology of pooling and also the poolability tests suggested by the committee. State specific pooled results are presented in Chapters 4 to 10 of Part-II of the manual, each chapter being dedicated to a state mentioned above. Each chapter from 4 to 10 is further organised in 3 sections. Section 1 provides sample size in terms of First Sage Units (FSU), Second Stage Units (SSU) and number of persons surveyed separately for each schedule type (Schedule 1.0, Type-1&2) and sample (central and state sample). Poolability tests of central and state sample in terms of MPCE (monthly per capita consumer expenditure) at district level are presented in Section 2. The final section viz. section 3 is dedicated to pooled results of schedule 1.0 type-1 &2 for estimated persons, sex ratio and MPCE (separately for Food, Non-food and Total group) along with RSE (relative standard error) of estimates and central Manual on Pooling of Central and State Sample Data Page 4

10 and state sample estimates at district and state level. Summary analysis of poolability tests and pooled results for Total MPCE are provided in chapter 11 of Part-II. Part-III of the manual contains the estimation procedure followed in NSS 66 th round for building estimates at central and state level (Annexure-I); State-wise, District-wise allocation of First Stage Units in Central and State Sample for Rural and Urban Sectors (Annexure-II and III respectively) and also the facsimile of Schedules 1.0, (Type-1&2) in Annexure-IV & V respectively. Manual on Pooling of Central and State Sample Data Page 5

11 Chapter2: Sample design and concepts and definitions Overview: The 66 th Round (July 2009-June 2010) of NSS was earmarked for survey on Household Consumer Expenditure and Employment and Unemployment. The survey on Household Consumer Expenditure and Employment and Unemployment was the eighth quinquennial survey in the series. The survey covered the whole of the Indian Union except (i) interior villages of Nagaland situated beyond five kilometres of the bus route and (ii) villages in Andaman and Nicobar Islands which remain inaccessible throughout the year. For Leh (Ladakh) and Kargil districts of Jammu & Kashmir there were no separate sample firststage units (FSUs) for central sample. For these two districts, sample FSUs drawn as state sample were treated as central sample. During this round, the following schedules of enquiry were canvassed: Schedule 0.0 Schedule 1.0 Schedule 10 : List of Households : Consumer Expenditure : Employment and Unemployment Two types of Schedule 1.0 viz. Schedule Type-1 and Schedule Type-2 were canvassed in this round. Schedule Type-2 had different reference period (7 days) for some items of food, pan, tobacco and intoxicants as compared to 30 days reference period for these items in Schedule Type-1. In this round all the States and Union Territories except Andaman & Nicobar Islands, Chandigarh, Dadra & Nagar Haveli and Lakshadweep participated. The following was the matching pattern of the participating States/ UTs. Nagaland (U) Uttar Pradesh, J & K, Manipur & Delhi Maharashtra (U) & Kerala Gujarat Remaining States/ UTs : triple : double : one and half : less than equal : equal Sample Design: A stratified multi-stage design had been adopted for the 66 th round survey. The first stage units (FSUs) were the 2001 census villages (Panchayat wards in case of Kerala) in the rural sector and Urban Frame Survey (UFS) blocks in the urban sector. The ultimate stage units (USUs) were households in both the sectors. In case of large FSUs, one intermediate stage of sampling was the selection of two hamlet-groups (hgs)/ sub-blocks (sbs) from each rural/ urban FSU. Sampling Frame for First Stage Units: For the rural sector, the list of 2001 census villages (henceforth the term village would mean Panchayat wards for Kerala) constituted the Manual on Pooling of Central and State Sample Data Page 6

12 sampling frame. For the urban sector, the list of latest available UFS blocks was considered as the sampling frame. Stratification: Within each district of a State/ UT, generally speaking, two basic strata were formed: i) rural stratum comprising of all rural areas of the district and (ii) urban stratum comprising of all the urban areas of the district. However, within the urban areas of a district, if there were one or more towns with population 10 lakhs or more as per population census 2001 in a district, each of them formed a separate basic stratum and the remaining urban areas of the district was considered as another basic stratum. Sub-stratification: There was no sub-stratification in the urban sector. However, to net adequate number of child workers, for all rural strata, each stratum was divided into 2 substrata as follows: sub-stratum 1: all villages with proportion of child workers (p) >2P (where P is the average proportion of child workers for the State/ UT as per Census 2001) sub-stratum 2: remaining villages Total sample size (FSUs): FSUs for central sample and FSUs for state sample have been allocated at all-india level. Allocation of total sample to States and UTs: The total number of sample FSUs was allocated to the States and UTs in proportion to population as per census 2001 subject to a minimum sample allocation to each State/ UT. While doing so, the resource availability in terms of number of field investigators was kept in view. Allocation of State/ UT level sample to rural and urban sectors: State/ UT level sample size was allocated between two sectors in proportion to population as per census 2001 with double weightage to urban sector subject to the restriction that urban sample size for bigger states like Maharashtra, Tamil Nadu etc. should not exceed the rural sample size. A minimum of 16 FSUs (to the extent possible) was allocated to each State/ UT separately for rural and urban areas. Further the State level allocations for both rural and urban had been adjusted marginally in a few cases to ensure that each stratum/ sub-stratum gets a minimum allocation of 4 FSUs. Allocation to strata/ sub-strata: Within each sector of a State/ UT, the respective sample size was allocated to the different strata/ sub-strata in proportion to the population as per census Allocations at stratum/ sub-stratum level were adjusted to multiples of 4 with a minimum sample size of 4. Selection of FSUs: For the rural sector, from each stratum/ sub-stratum, required number of sample villages were selected by probability proportional to size with replacement (PPSWR), size being the population of the village as per Census For urban sector, from each stratum FSUs were selected by using Simple Random Sampling Without Replacement (SRSWOR). Both rural and urban samples were drawn in the form of two independent subsamples and equal number of samples were allocated among the four Sub-rounds. Listing of households: Having determined the hamlet-groups/ sub-blocks, i.e. area(s) to be considered for listing, the next step is to list all the households (including those found to be Manual on Pooling of Central and State Sample Data Page 7

13 temporarily locked after ascertaining the temporariness of locking of households through local enquiry). The hamlet-group/ sub-block with sample hg/ sb number 1 were considered for listing first, to be followed by the listing of households within the sample hg/ sb number 2. Formation of second stage strata and allocation of households Two cut-off points A and B (in Rs.) were determined from NSS 61 st round data for each NSS region for urban areas in such a way that top 10% of the population have MPCE more than B and bottom 30% of the population have MPCE less than A. For both Schedule 1.0 and Schedule 10, households listed in the selected FSU/ hamlet-group/ sub-block were stratified into three second stage strata (SSS). Composition of the SSS and number of households to be surveyed from different SSS for each of the three schedules of enquiry namely, Schedule 1.0 (Type 1), Schedule 1.0 (Type 2) and Schedule 10 was as follows: SSS composition of SSS number of households to be surveyed FSU without hg/sb formation FSU with hg/sb formation (for each hg/sb) Rural SSS 1: relatively affluent households 2 1 SSS 2: of the remaining, households having principal earning from non- agricultural activity 4 2 SSS 3: other households 2 1 Urban SSS 1: households having MPCE of top 10% of urban population (MPCE > B) SSS 2: households having MPCE of middle 60% of urban population (A MPCE B) SSS 3: households having MPCE of bottom 30% of urban population (MPCE A) Selection of households: From each SSS the sample households for each of the schedules were selected by SRSWOR. If a household was selected for more than one schedule, only one schedule was canvassed in that household in the priority order of Schedule 1.0 (Type-1), Schedule 1.0 (Type-2) and Schedule 10 and in that case the household was replaced for the other schedule. If a household was selected for Schedule 1.0 (Type-1) it was not selected for Schedule 1.0 (Type-2) or Schedule 10. Similarly, if a household was not selected for Schedule 1.0 (Type-1) but selected for Schedule 1.0 (Type-2) it was not selected for Schedule 10. Manual on Pooling of Central and State Sample Data Page 8

14 Concepts and definitions: Concepts and definitions of some important terms used in this round and which are related to the results presented in this manual are given below. For detailed concepts and definitions of all other terms, the report of NSS 66 th round may be referred to. Household: A group of person normally living together and taking food from a common kitchen constitutes a household. The word "normally" means that temporary visitors are excluded but temporary stay-aways are included. Thus, a son or daughter residing in a hostel for studies is excluded from the household of his/her parents, but a resident employee or resident domestic servant or paying guest (but not just a tenant in the house) is included in the employer/host's household. "Living together" is usually given more importance than "sharing food from a common kitchen" in drawing the boundaries of a household in case the two criteria are in conflict; however, in the special case of a person taking food with his family but sleeping elsewhere (say, in a shop or a different house) due to space shortage, the household formed by such a person's family members is taken to include that person also. Each inmate of a mess, hotel, boarding and lodging house, hostel, etc., is considered as a single-member household except that a family living in a hotel (say) is considered as one household only; the same applies to residential staff of such establishments. Under-trial prisoners in jails and indoor patients of hospitals, nursing homes, etc., are considered as members of the households to which they last belonged. Household size: The size of a household is the total number of persons in the household. Household consumer expenditure: This is the expenditure incurred by a household on domestic consumption during the reference period. Expenditure incurred towards productive enterprises of households is excluded from household consumer expenditure. Also excluded are expenditure on purchase and construction of residential land and building, interest payments, insurance premium payments, payments of fines and penalties, and expenditure on gambling including lottery tickets. Money given as remittance, charity, gift, etc. is not consumer expenditure. However, self-consumed produce of own farm or other household enterprise is valued and included in household consumer expenditure. So are goods and services received as payment in kind or free from employer, such as accommodation and medical care, and travelling allowance excluding allowance for business trips. For articles of food (including pan, tobacco and intoxicants) and fuel, household consumption is measured by the quantity of the article actually used by the household during the reference period, irrespective of the expenditure incurred on it. For articles of clothing and footwear, consumption by a household is considered to occur at the moment when the article is brought into maiden or first use by any household member. The consumption may be out of (a) purchases made in cash or credit during the reference period or earlier; (b) home-grown stock; (c) receipts in exchange of goods and services; (d) any other receipt like gift, charity, borrowing and (e) free collection. Home produce is evaluated at the ex farm or ex factory rate. For evaluating household consumption of all other items, a different approach is followed: the expenditure made by the household during the reference period for the purchase or acquisition of goods and services, regardless of when the goods and services are used and by whom, is considered as household consumption. However, for a few items of expenditure such as rent, telephone charges, consumer taxes and railway season tickets, expenditure Manual on Pooling of Central and State Sample Data Page 9

15 during the month is recorded as the amount that was last paid divided by the number of months to which the payment related. To simplify data collection, consumption of food processed in the home from one item into another, such as milk converted into curd or butter, vegetables converted into pickles, and rice converted into liquor are recorded in the survey against the primary or ingredient item(s), such as milk, instead of the item in which form it is consumed (e.g. curd). For some item groups such as intoxicants, this procedure leads to an underestimation of consumption with a corresponding overestimation of the item groups of the major ingredients, such as cereals. Value of consumption: For items of food, pan, tobacco, intoxicants, fuel, clothing and footwear, this term is not synonymous with expenditure incurred by the household on the item, and the following rules of valuation are specified. Consumption out of purchase is evaluated at the purchase price. Consumption out of home produce is evaluated at ex farm or ex factory rate. Value of consumption out of gifts, loans, free collections, and goods received in exchange of goods and services is imputed at the rate of average local retail prices prevailing during the reference period. Monthly Per Capita (consumer) Expenditure (MPCE): This is defined as household consumer expenditure divided by household size. Uniform Reference Period MPCE (or MPCE URP ): This is the measure of MPCE obtained by the NSS consumer expenditure survey (CES) when household consumer expenditure on each item is recorded for a reference period of last 30 days (preceding the date of survey). Mixed Reference Period MPCE (or MPCE MRP ) This is the measure of MPCE obtained by the CES when household consumer expenditure on items of clothing and bedding, footwear, education, institutional medical care, and durable goods is recorded for a reference period of last 365 days, and expenditure on all other items is recorded with a reference period of last 30 days. Modified Mixed Reference Period MPCE (or MPCE MMRP ) This is the measure of MPCE obtained by the CES when household consumer expenditure on edible oil, egg, fish and meat, vegetables, fruits, spices, beverages, refreshments, processed food, pan, tobacco and intoxicants is recorded for a reference period of last 7 days, and for all other items, the reference periods used are the same as in case of Mixed Reference Period MPCE (MPCE MRP ). Manual on Pooling of Central and State Sample Data Page 10

16 Chapter 3: Pooling Methodology Testing poolability of central and state sample Though the central sample and state sample are drawn independently following identical sampling design with same concepts, definitions and instructions to collect the state sample data but due to lack of adequate training of field and processing staff of State DES, unit level data in some cases are not properly validated. There is also expected agency bias in the two sets of data generated by different agencies. As such they cannot be merged for generating pooled estimate without testing that the samples are realized from identical distribution function. Since the parametric distribution of the sample mean is unknown one may adopt non-parametric tests such Run test, Median test, chi-square test etc to test that the samples are coming from identical distribution function. Median test In statistics, the median test is a special case of Pearson's Chi-square test. It tests the null hypothesis that the medians of the populations from which two samples are drawn, are identical. Observations in each sample are assigned to two groups, one consisting of data whose values are higher than the median value in the two groups combined, and the other consisting of data whose values are at the median or below. A Pearson's Chi-square test is then used to determine whether the observed frequencies in each group differ from expected frequencies derived from a distribution combining the two groups. Let m * be the median of the pooled sample data. Construct 2 X 2 contingency table as below and use chi-square test if State sample and Central sample have identical median. Sample-type no of sample observation <= m * > m * Total State Sample N 11 N 12 N 1. Central Sample N 21 N 22 N 2. Total N.1 N.2 N.. Observed frequency of each cell O ij = N ij where i= 1 to 2, j= 1 to 2. Expected frequency of each cell E ij = (N i. * N.j )/N.. where i= 1 to 2, j= 1 to 2. 2 Value = 2 2 i 1 j 1 ( Oij 2 Eij ) / Oij with degrees of freedom = (2-1)*(2-1) = 1 The statistical power of this test may sometimes be improved by using a value other than the median to define the groups say quintile classes that is, by using a value which divides the groups into more nearly equal groups than the median would. Manual on Pooling of Central and State Sample Data Page 11

17 Multinomial distribution test or 2 test For discrete data such as status of activity, educational level and categorical variable such as land possed etc, standard tests of equality of sample proportions of two sets of data based on multinomial distributions, relevant chi-square tests may be used after grouping the attributes/categorical variables in to a suitable number of classes so that each class contains adequate number of sample observations. Construct 2 X k contingency table for k classes at the domain where two sets of data are to be pooled as below and use chi-square test if State sample and Central sample have identical distribution. Sample-type no of sample observation Class-1 Class-2... Class-k-1 Class-k Total State Sample N 11 N N 1k-1 N 1k N 1. Central Sample N 21 N N 2k-1 N 2k N 2. Total N.1 N.2... N.k-1 N.k N.. Observed frequency of each cell O ij = N ij where i= 1 to 2, j= 1 to k. Expected frequency of each cell E ij = (N i. * N.j )/N.. where i= 1 to 2, j= 1 to k. 2 Value = i 1 j 1 ( Oij Eij ) / Oij with degrees of freedom = (2-1)*(k-1) = k-1 Wald-Wolfowitz run test (non-parametric) Suppose X and Y are independent random samples with cumulative distribution function (CDF) as F s (x) and F c (y). Null Hypothesis to be tested is H 0: F s (x) = F c (x) for all x against alternative Hypothesis is H 1 : F s (x) <= F c (x) for all x and F s (x) < F c (x) for some x. Let x 1, x 2,.., x m be iid observation from state sample with distributive function F s and y 1,y 2,..,y n be iid observation from central sample with distributive function F c. Pool the data and order them with respect to comparable characteristic under consideration say monthly per capita expenditure (MPCE). In the pooled order sequence put 1 for X and 0 for Y. Let U be the total runs observed where 'run' is a sequence of adjacent equal symbols. For example, following sequence: is divided in six runs, three of them are made out of 1 and the others are made out of 0. The number of runs U is a Manual on Pooling of Central and State Sample Data Page 12

18 random variable whose distribution for large sample can be treated as normal with: 2mn 1 mean: m n variance: 2mn(2mn m n) ( m n) 2 ( m n 1) After normalizing the variable U one may use one sided z-test for testing the Null hypothesis. In extreme case the value of U will be 2 meaning by observed characteristic of all the observation of one sample is less than the other samples. One of the limitations of this test is when there is a tie between two samples in the observed value. One has to resolve ties in usual manner. However if there is large number of ties which is bound to occur specially for qualitative attributes like education level, activity status etc, this test is not recommended. This test can be well applied for a continuous variable such as MPCE which are less prone to ties. For discrete variable chi-square test is recommended. Parametric test Aggregate estimate: Let t yc and t ys be the estimate of Y at domain level of pooling based on central and state sample respectively with corresponding variances V(t yc ) and V(t ys ). For large sample, making all assumption of parametric test, one may use Z-Statistic to test the null hypothesis H 0 E(t yc ) = E(t ys ) where E stands for expectation. ( t yc Z= ( V ( t ) V ( t )) yc t ys ) ys V(t yc ) and V(t ys ) could be estimated as ^ V ( t yc ) l ( t yc1 t yc2 ) 2 / 4, ^ V ( t stands for summing over stratum x sub- based on sub-sample 1 & 2 estimates where l stratum level variance at the domain of pooling. ) Manual on Pooling of Central and State Sample Data Page 13 ys l ( t ys1 t ys2 ) 2 / 4

19 Estimated rates/ratios: Let r c and r s be the estimate of population rates R c and R s ie Y/X based on central and state sample respectively with corresponding mean square error MSE(r c ) and MSE (r s ). For large sample, making all assumption of parametric test, one may use Z-Statistic to test the null hypothesis H 0 E(r c )=E(r s ) where E stands for expectation. Z= ( r c ( MSE( r c r s ) ) MSE( r s )) MSE(r c ) and MSE(r s ) are estimated as follows: mse(r c ) = ( ^ V (tyc ) 2 * r c ^ Cov (t yc, t xc ) + r c 2 * ^ V (t xc ))/ t xc 2 mse (r s ) = ( ^ V (tys ) 2 * r s ^ Cov (t ys, t xs ) + r s 2 * ^ V (t xs ))/ t xs 2 where ^ V ( t yc ) l ( t yc1 t yc2 ) 2 / 4 ^ V ( t ys ) l ( t ys1 t ys2 ) 2 / 4 ^ V ( t xc ) l ( t xc1 t xc2 ) 2 / 4, ^ V ( t xs ) l ( t xs1 t xs2 ) 2 / 4 Manual on Pooling of Central and State Sample Data Page 14

20 ^ Cov (t yc, t xc ) = based on sub-sample 1 & 2 estimates. l l ( tyc1 t yc2)( txc1 txc2 ) / 4 where stands for summing over stratum x sub-stratum level variance, covariance at the domain of pooling. Methodology for pooling Pooling by inverse weight of the variance of the estimates Aggregate estimate: For any characteristic, consider the state sample [s] in the form of two independent sub- sample s1 and s2 and the central sample [c] in the form of two independent sub- sample c1 and c2. Based on this, the respective estimates for state and central can be computed as: t s = l (t s1 + t s2 )/2 and t c = l (t c1 + t c2 )/2 Pooled estimate leading to optimum combination of these two estimates is given by weighing with inverse of the variance of the estimate. Thus the pooled estimate is given by: T p = V(T p ) = V ( tc) ts V ( t V ( t ) V ( t c In general V t ) and ( c s s ) t ) c V ( tc) V ( ts) V ( t ) V ( t ) c V ( t s ) s with are unknown and can be estimated as ^ V ( t c ) l ( t c1 t c2 ) 2 / 4, Manual on Pooling of Central and State Sample Data Page 15

21 Manual on Pooling of Central and State Sample Data Page 16 4 / ) ( ) ( ^ s s l s t t t V where l stands for summing over stratum x sub-stratum level variance at the domain of pooling. Thus pooled estimate and estimate of pooled variance is given by t p = ) ( ) ( ) ( ) ( ^ ^ ^ ^ s c c s s c t V t V t t V t t V, ) ( ^ t p V = ) ( ) ( ) ( ) ( ^ ^ ^ ^ s c s c t V t V t V t V By virtue of weighing the two estimates at the domain level at which two estimates are pooled, the pooled estimate will always lie between the central and state sample estimates. Estimated rates/ratios: Let r c and r s be the estimate of R c and R s ie Y/X based on central and state sample respectively with corresponding estimated mean square error mse(r c ) and mse(r s ). The pooled estimate and estimate of variance of pooled ratio estimate may be given by: r p = ) ( ) ( ) ( ) ( s c c s s c r mse r mse r r mse r r mse,

22 mse( r p ) = mse( r mse( r c c ) mse( r s ) ) mse( r s ) Where mse(r c ) and mse(r s ) are calculated using formula given in para above. Alternatively one can generate the pooled estimate of aggregate by inverse weight of estimate of variance obtained from central and state sample using formula given in para for the characteristics x as well as y and obtain the pooled estimate of ratio as ratio of pooled estimate of aggregate. This will ensure consistency between pooled estimates of aggregate and the pooled estimate of ratio. Let t xp and t yp be the pooled estimate of aggregate for the parameter X and Y. The pooled estimate of R (i.e Y/X) is given by r p= t yp / t xp where t yp = at yc + bt ys and t xp = ct xc + dt xs and (a, b), (c, d) are the estimated inverse variance weight pair of the characteristic x and y respectively. The estimated mse of pooled ratio estimate r p is given by: mse(r p ) = ( ^ V (typ ) 2 r p ^ Cov (t yp, t xp ) + r p 2 ^ V (t xp ))/ t xp 2 where ^ V ( t yp ) ab = a b, ^ V ( t xp ) cd = c d and ^ Cov (t yp, t xp )= ac Cov ^ ( t yc, t xc ) +bdcov ^ ( t ys, t xs ). ^ Cov (t yc, t xc )= Similarly, l where of pooling. ^ Cov (t ys, t xs )= l ( tyc1 t yc2)( txc1 txc2) / 4 l ( t ys1 tys2)( txs1 txs2) / 4 based on sub-sample 1 & 2 estimates. stands for summing over stratum x sub-stratum level covariance at the domain Manual on Pooling of Central and State Sample Data Page 17

23 Method laid down in para and requires calculation of estimate of variance of the estimates before pooling them. Reliability of estimate of variance should be ascertained with due consideration of sample size. Besides the complex calculations of variances and covariances for each cell of the table, one needs to address the issue of non-additivity of the component estimates with the estimate of marginal total. For e.g. pooled estimate of MPCE of FOOD and NON-FOOD may not add up to MPCE of TOTAL. To obviate this problem one may generate the pooled estimates of components first and then derive the estimate of total as sum of estimates of components. Pooling by simple average of the estimates Many of the States are not fully equipped with complex calculation of estimate of variance especially when cells of the table contains ratio of two characteristics which is usually presented in the NSS reports. When the State s participation is equal matching of central samples, the simple average of two estimates may be a way of combining the estimates considering central and state samples as independent samples. The pooled estimate will always lie between the estimates based on central and state sample separately. Pooling by weightage average of the estimates by matching ratio of states participation When the State s participation is of unequal matching of central samples, the weighted average of two estimates with weights being matching ratio of central and state sample may be a better way of combining the estimates considering central and state samples as independent samples. For any characteristic, consider the state sample [s] in the form of two independent sub-sample s1 and s2 and the central sample[c] in the form of two independent sub- sample c1 and c2. Let matching ratio of state and central sample be m : n. Based on this, the respective estimates for state and central can be computed as: t s = l (t s1 + t s2 )/2 and t c = l (t c1 + t c2 )/2 Pooled estimate of these two estimates is given by weighing with matching participation rate m:n. Thus the pooled estimate is given by: t p = mts m In general ^ V ( t s ) nt n c V ( t c ) V ( t and s l ( t s1 t s2 2 2 m V ( ts ) n with V(t p ) = ( m n) ) 2 / 4 ) can be estimated as and thus ^ V ( t p ) V ( t Manual on Pooling of Central and State Sample Data Page 18 m ^ V ( t 2 ^ c 2 ) l n c ) ( t 2 c1 ^ t V ( ts ) V ( tc) 2 ( m n) = c2 ) 2 / 4,

24 l where of pooling. stands for summing over stratum x sub-stratum level covariance at the domain The pooled estimate will always lie between the estimates based on central and state sample separately. Summing up: For those characteristics which are known to be distributed as Normal, poolability of the two sets of central and state data may be tested by standard parametric tests such as Z-test. For those characteristics for which transformation makes them Normal, such methodology may be adopted. In most of the situations where the distribution is non-normal and unknown, the two sets of data may be tested through various non-parametric tests such as those laid down above. For discrete data, Standard tests of equality of proportions based on binomial distribution may be used and for multinomial distributions relevant chi-square tests may be used. Manual on Pooling of Central and State Sample Data Page 19

25 Part-II State Results (Andhra Pradesh, Bihar, Gujarat, Himachal Pradesh, Karnataka, Kerala, Odisha)

26 Pooling exercise in respect of seven states viz. Andhra Pradesh, Karnataka, Kerala, Bihar, Odisha, Himachal Pradesh and Gujarat for the Schedule 1.0, consumer expenditure of NSS 66 th round have been so far completed following the methodology suggested by the NSC committee and results of these seven states are presented in this section of the manual. Pooling exercise in respect of Bihar and Odisha was done by DPD based upon validated data supplied by these two states. There were some discrepancies noticed in the estimates provided by DES Gujarat for MPCE (non-additivity of Food and non-food MPCE to total MPCE) and RSE of central sample and pooled sample accordingly estimates were regenerated at DPD based upon validated data supplied by the state along with the pooled results. It was noticed that state of Gujarat participated in NSS 66 th round with less than equal matching for some districts. Pooled estimates for this state by matching ratio were worked out with equal weight. Although the state DES of Delhi has also carried out the exercise of pooling for 66 th round, pooling has been done at state level and not at district level. Rural sector in Delhi is very small in size and entire rural area was made a single stratum in the 66 th round. For urban sector, Delhi Municipal Corporation was made one stratum and all other towns were made another stratum in NSS 66 th round. Moreover, UFS frame of Delhi does not contain the district of all the towns. Owing of these reasons, district level estimates for Delhi could not be attempted and hence the results of Delhi are not presented in this manual. The basic requirements for pooling are that State and Central sample data should have common layout and passed through common validation checks and the data sets should be poolable. Non sampling error must be removed from two sets of data after following the uniform frame work both at data collection stage and data processing stage so that two sets of data are poolable. NSSO follows rigorous procedure right from data collection through the mechanism of two tier training, inspection, supervision, field scrutiny and different stage of validating data entered at DPD. Working Group constituted for each round by NSC also examines the final results brought out by NSSO. Though state samples are drawn with identical sampling design, infrastructure and institutional setup at state DES varies from state to state. Quality of data collected by the state agency depends upon extent of training provided to field functionaries. Similarly extent of data validation depends upon resource availability in the state DES. States are shared with data processing instruments followed in central sample processing however onus lies with state DES for adopting the same for state sample in uniform manner. Unlike NSSO there is no mechanism to vetting the results derived from state sample data. Thus the onus on validity of pooled results presented in this section of the manual lies with the state DES and the results presented here is just compilation of results brought out by the state DES based upon two sets of data for the benefit of users. There is no mechanism to validate the state sample data at DPD. As a precautionary measure some sample checks of estimates based upon central sample data was carried out at DPD before placing the results in the manual. Thus the results are to be used cautiously by taking into cognizance of poolabilty tests performed and RSE of estimates and sample size allotted at district level.

27 Chapter 4 Pooled Results of Andhra Pradesh

28 Section-I: Sample Size Total sample size of Andhra Pradesh for central and state sample are given below: ANDHRA PRADESH RURAL Central sample State sample Schedule FSU allotted HH surveyed Persons surveyed FSU allotted HH surveyed Persons surveyed 1.0 Type-I Type-II ANDHRA PRADESH URBAN Central sample State sample Schedule FSU allotted HH surveyed Persons surveyed FSU allotted HH surveyed Persons surveyed 1.0 Type-I Type-II AP.2

29 Section-II Poolability Test State: Andhra Pradesh Sector: RURAL [SCHEDULE 1.0 TYPE-I AND TYPE-II] RUN TEST TABLE-0.1 (R): DISTRICT-WISE RESULTS OF RUN TEST OF MPCE (URP,MRP,MMRP) FOR POOLED SAMPLE Z 0.01 = {one sided test} reject if z-value <Z 0.01 Dist District Name URP MRP MMRP Z- VALUE Accept Z-value Accept Z-value Accept 1 Adilabad Yes Yes Yes 2 Nizamabad Yes Yes Yes 3 Karimnagar Yes Yes Yes 4 Medak Yes No Yes 6 Rangareddy No No No 7 Mahabubnagar No No No 8 Nalgonda No Yes No 9 Warangal Yes Yes Yes 10 Khammam No Yes Yes 11 Srikakulam Yes Yes No 12 Vizianagaram 0 Yes Yes No 13 Visakhapatnam No Yes Yes 14 East Godavari Yes Yes Yes 15 West Godavari Yes Yes Yes 16 Krishna Yes Yes Yes 17 Guntur 0 Yes Yes Yes 18 Prakasam Yes Yes Yes 19 Nellore Yes Yes Yes 20 Kadapa Yes Yes Yes 21 Kurnool No Yes No 22 Anantapur Yes Yes -2.2 Yes 23 Chittoor Yes Yes Yes Andhra Pradesh Yes Yes Yes AP.3

30 State: Andhra Pradesh Sector: URBAN [SCHEDULE 1.0 TYPE-I AND TYPE-II] RUN TEST TABLE-0.1 (U): DISTRICT-WISE RESULTS OF RUN TEST OF MPCE (URP,MRP,MMRP) FOR POOLED SAMPLE Z 0.01 =-2.33 {One Sided Test} Reject If Z-Value <Z 0.01 Dist District Name URP MRP MMRP Z- VALUE Accept Z-value Accept Z-value Accept 1 Adilabad Yes Yes Yes 2 Nizamabad Yes Yes Yes 3 Karimnagar Yes No Yes 4 Medak Yes Yes No 5 Hyderabad Yes No Yes 6 Rangareddy Yes Yes Yes 7 Mahabubnagar Yes Yes Yes 8 Nalgonda Yes Yes Yes 9 Warangal Yes Yes Yes 10 Khammam Yes No Yes 11 Srikakulam 0 Yes Yes Yes 12 Vizianagaram Yes Yes Yes 13 Visakhapatnam Yes Yes Yes 14 East Godavari Yes Yes Yes 15 West Godavari Yes Yes Yes 16 Krishna No Yes Yes 17 Guntur Yes Yes Yes 18 Prakasam Yes Yes Yes 19 Nellore Yes Yes Yes 20 Kadapa Yes Yes Yes 21 Kurnool Yes Yes Yes 22 Anantapur Yes Yes Yes 23 Chittoor Yes Yes Yes Andhra Pradesh Yes Yes Yes AP.4

31 State: Andhra Pradesh Sector: RURAL [SCHEDULE 1.0 TYPE-I AND TYPE-II] MEAN TEST TABLE-0.2 (R): DISTRICT-WISE TEST OF MPCE DIFFERENCE (URP,MRP,MMRP) FOR POOLED SAMPLE Z = {two sided test} reject if absolute z-value >Z Dist District Name URP MRP MMRP Z- VALUE Accept Z-value Accept Z-value Accept 1 Adilabad Yes Yes Yes 2 Nizamabad Yes Yes Yes 3 Karimnagar Yes Yes Yes 4 Medak No Yes Yes 6 Rangareddy Yes No No 7 Mahabubnagar No Yes Yes 8 Nalgonda No No No 9 Warangal No No Yes 10 Khammam No Yes No 11 Srikakulam Yes Yes Yes 12 Vizianagaram Yes Yes No 13 Visakhapatnam No No No 14 East Godavari Yes Yes Yes 15 West Godavari Yes Yes Yes 16 Krishna Yes Yes Yes 17 Guntur Yes Yes No 18 Prakasam No No Yes 19 Nellore Yes Yes Yes 20 Kadapa Yes Yes Yes 21 Kurnool Yes Yes Yes 22 Anantapur No Yes Yes 23 Chittoor No No Yes Andhra Pradesh Yes Yes Yes AP.5