Abstract

Robust statistical data forms the cornerstone of an informed governance system. This paper studies the statistical system and data dissemination in the Centre and State governments in India, and the measures put in action to accelerate the data dissemination process. Arguing that the availability of high-frequency statistical data is a necessary condition for good governance, the first section of the paper reflects on the structural and functional framework of the Indian Statistical System, and how instrumental the various government ministries have been in introducing and facilitating the increasing availability of data for public welfare and information. Arguments are drawn from closely studying governmental initiatives about Open Government Data (OGD) and other policies that form the nucleus of data transparency and dissemination. The second section discusses the role of State governments in data collection and distribution vis-à-vis Centre-State relations under the Indian data governance profile. Substantial data asymmetry has been identified among the States which is further explained within the framework of Panchayati Raj and grass-root governance. The last section of the paper focuses on the limitations prevalent in statistical governance and provides holistic recommendations to amend the existing gaps in the system.

Introduction

This chapter is set in context to Indian statistical data particularly, and how functional the central and the state governments have been about data dissemination for public transparency and better governance since the official statistical system forms an important part of a country's institutional framework. The centrality of data for governance is reflected in the origin of the word ‘statistics’ which denotes a crucial role in managing state affairs. Credible data provides a shared understanding between the government and its citizens about the understanding of the reality of the state of affairs in a country (Muralidharan). Data dissemination in Government is primarily releasing information obtained through recorded data for its users, to ensure smooth governance. Statistical data, to define, refers to the collection, organization and analysis of information to make inferences, decisions, or conclusions about a particular phenomenon or population.

Under the able leadership of P.C. Mahalanobis who is also regarded as the pioneer of the Indian Statistical System India set up the Central Statistical Unit in 1949, which was later converted into the Central Statistical Organization (CSO) in 1951. Alongside this, the Department of Statistics was also established, which presently serves as the Statistics Wing of the Ministry of Statistics and Programme Implementation (MoSPI). High-quality data allows the government to understand problems of development, which was imperative for a country like India right after independence. At present, the Ministry of Statistics and Programme Implementation (MoSPI) is the nodal ministry of the Government of India (GoI) concerned with aspects of coverage and quality of statistics that are regularly released (Kumar) and the surveys conducted by the Ministry are based on scientific sampling methods. To understand the statistical system of India, the understanding of the functioning and distribution of administrative powers between the Centre and the states is important. The Union, State, and Concurrent Lists, as specified in the Indian Constitution, serve as the basis for the administrative function division between the Central Government of India and the State Governments. As per the Allocation of Business Rules, 1961, which are subject to periodic modifications, the duties are distributed among the different ministries and departments at the Centre. Any authority (Central Ministry or Department or State Government Department) that is in charge of that subject based on its listing in the Union, State, or Concurrent Lists is typically in charge of gathering statistics on that subject (Ministry of Statistics and Programme Implementation).

Assessing the Role of the Central Government in Data Dissemination

Statistical data is a public asset, both in terms of accountability of the elected representatives and enabling a deeper understanding of the distribution of public goods and services. The Central government plays a crucial role in collecting and disseminating economic data in India. Government bodies like the MoSPI, Ministry of Labour, NITI Aayog, and the Reserve Bank of India (RBI) collect and manage a major part of economic data. In 2012, the Union Ministry of Science & Technology introduced the “National Data Sharing and Accessibility Policy (NDSAP)” to facilitate access to Government of India-owned shareable data (along with its usage information) in machine-readable form through a wide area network all over the country. To enable this, the Open Government Data (OGD) Platform (www.data.gov.in), which is an integrated repository of datasets, documents, services, tools and applications of various Ministries/Departments was launched in October 2012 for public use (Ministry of Statistics and Programme Implementation).

Keeping in terms with the spirit of a welfare state whose fundamental characteristic is inclusivity, India has always strived towards inclusive development instead of unchecked development, with the intention to empower the people at the bottom of the pyramid by raising their income and attempting the overall welfare of the society. As explained above, administrative powers are distributed between the Centre and the States, with certain subjects falling under the concurrent list. According to the Ministry of Statistics and Programme Implementation, data usually flows from the States to the Centre, except for subjects where state-level operations are integral to the centrally-sponsored schemes. By and large, state-wise results flow to the Centre, which is then aggregated to obtain national-level data on a particular subject. However, large-scale data collection of the nationwide population rests with the Centre, including the Population Census, Economic Census, Annual Survey of Industries and the like Ministry of Statistics and Programme Implementation.

In 1999, shortly after India underwent liberalisation and opened its markets to the world, the need for a stronger and more open public data system was felt. A new data-dissemination policy was introduced by the Union Ministry with the major objective of collecting data from various sources and integrating them into logical subject areas in such a way that the information available can be comprehended and used by non-technical decision-makers in the market as well as the common masses, thus making way for inclusive and accountable governance. The need for converting raw data into reports and the introduction of query tools to facilitate a better understanding of key challenges of the specific industries as well as the national problems was also suggested by various committees, to keep up with the market forces of the other countries (Bhattacharya). This project of creating a data warehouse became difficult to realize for the MoSPI and while the RBI became one of the earliest central banks to build a centralised system for economic data for the country, the MoSPI failed to reposition itself into, what can be called, the government’s analytical hub (Bhattacharya).

In the absence of any centralised data system for assembling statistical data of the country, individual industries went on to create their portals to share data and the need for integration of these data sets was felt yet again by late 2000. This led to the spearheading of a new data policy by the Ministry of Science and Technology in 2012 whose approval finally led to the introduction of India’s nascent open data system. This is not to say that this was the only factor leading to an open data system in the country. The introduction of the Right to Information (RTI) in 2005 had the provision of releasing suo-moto non-sensitive information to the public. With the help of the National Data Sharing and Accessibility Policy and the act's provisions, an Indian Civic Community of open data enthusiasts persuaded local, state, and federal government representatives to strive towards opening more datasets to the general public. Through their campaigning, some data flow was created for the recently established data.gov.in site, which was intended to be a single, unified location for viewing public data. Although data consumers' original expectations were not met by the data.gov.in portal, it did encourage government departments to make more datasets available to the public. As of 2020, the public's interest in collective data that has been suitably anonymized and secured has been acknowledged by the Kris Gopalakrishnan Committee Report on Non-Personal Data Governance Framework (Saxena). To facilitate safe data-driven development and provide access to non-personal data, it proposed a data-sharing architecture. The proliferation of the internet and digitalization in India has equipped numerous private and non-profit entities with the necessary infrastructure and capabilities to gather, store, and evaluate dynamic real-time data that can be beneficial for governance.

One of the most prominent areas where Open Government Data (OGD) has been instrumental is in addressing the monumental problem of data inaccuracy in India. (Write et al.,) mentions that while the RTI enables the dissemination of public data and information to the public, it does not guarantee the accuracy of data. Several civil society organisations and individuals have heavily complained about unreliable public data and according to the 2018 World Economic Report, India accounts for the most unequal country in terms of income, second to Russia (Kumar et.al.). This is a strong reflection of the failure of several welfare policies carved out by the Indian government over the years to combat inequality and disparity among various socio-economic classes present in the country. One of the main reasons for this is the insufficient data which governs these developmental schemes. Muralidharan explains how often bureaucrats and governments take a “blind shot” at designing public goods and services since they are mostly oblivious to the ground realities of the citizens and their actual needs. Another reason is the unavailability of actionable data. Input data denotes the distribution of policies or aids, but the impact made from it and people’s lived experience of those services mostly go undocumented, which accounts for actionable data. While there is just sufficient input data, there is not enough actionable data to analyse the impact of the welfare or developmental schemes, resulting in a vicious loop that lacks feedback from its very own users. of government departments and organisations. In many cases, not only extracting data is a difficult feat to achieve for non-governmental organisations and civil society, but examining the authenticity and reliability of data becomes an additional task. This is where Open Government Data (OGD) can be of great help. For starters, (Write et al.), mention again that a comparison of open data sets across websites and departments will result in easier cross-checking of data for accuracy. In actuality, access to government data is sometimes provided through digital means, but at a cost. This applies to data that ranges from corporate records kept up to date by the Ministry of Corporate Affairs (MCA21 MMP) to the Indian e-Gazette, which is the official source of all national legislation as well as the electronic form of the official repository for all notifications from the central government. Although certain fundamental data from the National Sample Survey is freely accessible, individuals must pay for any significant research. The Ministry of Electronics and Information Technology (MetiY) introduced the India Data Accessibility and Use Policy in 2022 intending to “radically transform India’s ability to harness public sector data” for large-scale transformation in society (Gupta). Drawing on a rather critical yet urgent need of the hour, the policy envisages sharing and maximizing access to all sorts of non-personal data with the help of existing open data portals as well as inter-government data sharing (Desk & Desk). Under this policy, the India Data Council (IDC) was also introduced which comprises the India Data Officer and Chief Data Officers of both the state and central governmental departments. In its official policy, it laid down the provisions for open data, stating that all governmental/departmental/ organisational data should have public access unless it is classified or has personal information of its users. They have also facilitated an extensive framework for government-to-government data sharing, high-value data sets, integrated data portals as well as data anonymization and privacy preservation (Ministry of Electronics and Information Technology, Government of India).

The Digital India Initiative, another flagship program of the GoI, was introduced in 2015 to transform India into a knowledge economy and digitally transformed society. The main aim of the initiative is to promote the digital literacy of the citizens and make digital data accessible to the public, thereby catapulting transparent governance and making digital resources accessible worldwide. Juxtaposing digital literacy and data dissemination, data accessibility and aware citizens are two direct results of this program (Welcome to Common Services). Although this initiative brings with it, its share of criticisms against the digital divide and asymmetry in its functionality, one timeline which has proven beneficial is during the COVID-19 pandemic. Garg et al., write about Participatory Disease Surveillance (PDS) which is a tool used for the self-reporting of communicable diseases by citizens to ensure proper statistical data to analyse on public health as well as timely intervention of health experts. With the provision of real-time capturing of data, PDS in India during the COVID-19 pandemic not only ensured to mapping of the location of the diseased but also served as a constructive platform to cross-check data with the government of laboratory-confirmed cases of the virus.

The official document of the National Statistical Office (NSO) of the Five-Year Vision 2019-2024 envisages the implementation of real-time monitoring of the Indian economy by using metadata, policymakers, ensuring better investment in Research and Development (R&D), Human Capital Management, increased user engagement and the like. It is indeed a matter to gauge on how the NSO brings in the functional changes to ameliorate the problems that have been bugging the process of transparent data collection and dissemination process.

The State Statistical System at the State level is a near replica of the system at the Centre. It follows the pattern of a laterally decentralised system divided into major Departments under the various Ministries of the State Government. Data collection, compilation, processing and preparation of results are carried out by the states for most sectors, of which agriculture and labour are of primal importance Ministry of Statistics and Programme Implementation and the Centre uses state-wise results for all-India-level Statistics. At the apex level lies the Directorate of Economics and Statistics (DES) which is formally responsible for coordinating the statistical activities in the State (Ministry of Statistics and Program Implementation) The DES displays somewhat consistent statistical activity. They calculate the estimates of the State Domestic Product and Retail Price Index Numbers, publish statistical abstracts and handbooks of the States, annual economic reviews or surveys, district statistical abstracts, and State budget analyses, among other statistical activities pertinent to the State. India has mostly standardised the statistical work programme of the States' DESs. “Their publications include state-specific statistical handbooks and abstracts, yearly economic reviews and surveys, budgets-in-brief, budgets classified by economic and functional factors, estimates of the state's gross domestic product, district statistical abstracts, development indicators of social and economic aspects, municipal yearbooks, and reports derived from the tabulation and analysis of sample survey data, including the NSS matching sample survey” (https://desagri.gov.in/en/). The State Statistical System is majorly concerned with micro-level planning and organisation as the policies and services born out of this data concern only a particular population, geography and concern. This data, is therefore, more crucial in the sense that proper collection and dissemination of it can not only lead to the suitable formation of state policies but also help draw an accurate picture of national problems and their challenges.

However, the State Statistical System (SSS) is plagued with problems and asymmetry. Democratic decentralisation was formally established in India in 1992 with the advent of the Panchayati Raj System, by introducing the 11^th and 12^th Schedules of the Indian constitution. These two schedules were born out of the 73^rd and 74^th Constitutional Amendment Act(s) that specify the powers and function of the Panchayats and Municipalities. However, even before that, the Block Statistical Organisation (BSO) was created almost fifty years ago to ensure statistical data for community development programs. The BSO enabled the establishment of statistical units under the supervision of the DESs on the lowest possible rung of the administrative ladder. Still, to much dismay, the work done by the BSO remains largely unimpressive, with very little coordination between the DESs and the BSO (Barman). Although the introduction of Local Self-Governance created ample opportunities to revive the BSO, much work has not been done to ensure its functionality and usefulness. The lack of a clear roadmap for navigating the challenges of data collection and distribution from the grassroots level to the concerned State and Union Ministries remains a struggle. The ODG portal of the GoI is highly suggestive of data asymmetry across the States. While Telangana is the only state with a dedicated ODG portal, most states showcase a rudimentary template of data access (Saxena).

Another arena where the statistical system forms the central requirement is mapping the rural-urban divide and the welfare policies encircling it. Government intervention is essential for the redistribution of income when it comes to the welfare of the poor. According to Rawls's (1971) difference principle, a just society should place its policies in such a way that it benefits the least well-off section of the society, that is, the welfare of the least fortunate group should be prioritized by society. (Sen) has explored poverty, inequality, and entitlement in great detail, both theoretically and empirically. The government has taken a leading role in policing for the betterment of the underprivileged. Two well-known examples of policy initiatives on inclusive growth are poverty alleviation programs like the Pradhan Mantri Jan Dhan Yojana (PMJDY) and the Mahatma Gandhi National Rural Employment Guarantee Act (MGNREGA). These are admirable initiatives aimed at maximizing growth benefits for economic well-being. However, these schemes rely heavily on data and play an important role in calculating the Gross Domestic Product (GDP) of the country as well. With time, government departments have fully computerised their data, enabling them to capture and digitalise highly disaggregated data. The Rangarajan Committee (2001) also recommended data consolidation using data warehousing technology. Besides, it also recommended the appointment of a professional statistician as the secretary of MoSPI, which was duly implemented. Despite these changes, the Indian Statistical System was hit by a major blow circa 2015 when the country's annual GDP figures did not match the other economic indicators present. The main argument was that the revision in the economic barometer that changed was not transparent enough. (The wire) The MCA-21, an e-governance initiative introduced by the Ministry of Corporate Affairs (MCA) was used to record company filings. Still, the way formal company filing proxies were used to calculate informal sector growth was announced faulty. This was followed by MoSPI’s refusal to open up its methodological practices and database for public scrutiny remained an obstruction to verify the gaps in the statistical dissemination framework in India (Bhattacharya).

Limitations/Suggestions

It has been well-established that India does not lack adequate policies and governmental bodies to overlook its statistical system. The presence of the MoSPI, NSO, NDSAP, DES, OGD platforms and the like are enough to testify about India’s growing concern about data transparency and accessibility. The problem remains elsewhere. Despite adequate measures being in place, India’s home and global performance has remained under par. One of the many reasons is the political tensions back home, especially between the Centre and the states. Political instability during the 1990s severely affected the statistical governance goals followed by several tensions of a coalition government, to date. The need of the hour is to establish proper channels of coordination among states so that high-frequency data reaches the designated portals/departments for use and analysis. Officers and staff members of all government departments should be imparted regular training on data collection and storage, required technical skills and above all, awareness about the importance of statistical data on the institutional framework of a country.

The second limitation lies in technological deficiencies. Despite making magnificent advancements in other sectors, India has failed to realise its technological potential in this sector. Government departments need to make full use of emerging technologies, especially Artificial Intelligence (AI) in capturing and storing data. The abundant availability of cheap internet can facilitate digital data collection, and proper use of AI and remote sensing can enable real-time data collection and analysis without the hassle of personnel being physically present at every site, in certain cases.

Thirdly, there is an urgent need for active citizen participation in the statistical framework process. The government should join hands with private organisations, civil society, NGOs and self-help groups to collect data. The active collaboration between organisations especially with proper reach in the rural parts of India and the government will draw benefits in assimilating quality data from the grass-roots. Besides, outsourcing the data collection and management process to a third party will have two direct benefits: it will act as a good source of cross-checking of data with the government departments and secondly, limited intervention of the concerned departments will obstruct the risk of under simplifying the problems recorded. The advent of several voluntary data reporting practices by concerned citizens has opened up more platforms for the government to strengthen its data sets.

Lastly, the census of India was last conducted in 2011, which needs to be conducted as more than a decade has passed. Consequently, a vicious cycle of data production delays and the ensuing accumulation of the most recent data backlogs is created. The private sector is far more prepared and producing data faster than the public sector in this era of digital information (Bhattacharya). This has made the argument between the state and the market even stronger. India is the only country with this kind of bureaucratic complexity; no other developed or emerging economy has it. In this era of PPP, as India makes its way into the top five economies of the world, it must take a serious look at the structural and functional setup of its statistical system, since it forms the base of its economy.

India: Data Governance Profile

1.Vision And Policy Intent

Digital India is a flagship programme of the Government of India with a vision to transform India into a digitally empowered society and knowledge economy.

The National Institution for Transforming India (NITI) Aayog's Strategy for New India includes the vision of “data led governance and policy making” with all policy interventions and decision- making based on evidence and real-time data. This requires:

•     Collecting data for new measurable parameters using latest technologies;

•     Improving efficiencies in processes related to existing data collection by government departments and agencies;

•     Expanding warehousing facilities for storing and integrating data;

•     Making data available for industry practitioners, academicians, researchers, etc., wherever feasible; and

•     Integrating data analysis and interactive data visualization into all policy formulation.

Aspirational Districts and Blocks programme, a flagship initiative of NITI Aayog, relies on real-time data collection for ranking indicators to foster competition, accountability and providing targeted support.

2. Data Management: Policies rules & institutionManagement:

The National Statistical Commission was constituted with effect from 12th July 2006 with a mandate to evolve policies, priorities and standards in statistical matters.

National Data Sharing and Accessibility Policy (NDSAP).

The National Policy is expected to increase the accessibility and easier sharing of non-sensitive data amongst the registered users and their availability for scientific, economic and social developmental purposes.

Department of Science & Technology (DST) would be the Nodal Department for all matters connected with overall co-ordination, formulation, implementation and monitoring of the policy.

The Ministry of Electronics and Information Technology, (MEITY) is working towards the promotion of e-Governance. MEITY has published a draft National Data Governance Policy which proposes the establishment of an “India Data Management Office” responsible for developing rules, standards, and guidelines for non-personal data to catalyse the data and AI based innovation ecosystem under this Policy.

3. Data Custodians and data stewardship

The Collection of Statistics Act 2008 is the principal legislation for the collection of economic, social, demographic, scientific and environmental data.

The Central Statistical Office (CSO) is the statistical arm of Ministry of Statistics and Programme Implementation (MoSPI). Its functions include:

•     Acting as the nodal agency for planned development of the statistical system.

•     Coordinating the statistical work in respect of the Ministries/Departments of the Government of India and State Statistical Bureaus (SSBs),

•     Advising the Ministries/ Departments of the Government of India on statistical methodology and on statistical analysis of data.

•     Compiling and releasing official statistics.

•     Organizing and conducting periodic all-India Economic Censuses and large-scale all- India sample surveys.

•     Evaluation of sampling design in respect of surveys conducted by the National Sample Survey Organisations and other Central Ministries and Departments.

•     Liaison with international statistical organizations.

4. Data sharing, accessibility and Integration

The National Data and Analytics Platform (NDAP) streamlines data

dissemination and provides access to standardized published government data sets from across India's extensive administrative landscape ina user-centric manner. NDAP allows users to easily search, merge, visualize, and download datasets.

To implement the National Data Sharing and Accessibility Policy (NDSAP), an Open Government Data (OGD) Platform has been developed by the National Informatics Centre, Ministry of Electronics & Informational Technology. The Open Government Data platform provides open access by proactive release of the data available with various ministries/departments/ organizations of Government of India. All datasets/resources including metadata published on the Open Government Data portal (data.gov.in) are licensed under the Government Open Data License-India which can be accessed here.