How Lanka Data Net’s automated data harvesting infrastructure is laying the foundation for the country’s next generation of artificial intelligence and digital public infrastructure
Colombo, Sri Lanka — In today’s digital economy, information is often described as the world’s most valuable resource. Yet despite the rapid digitisation of government services and public institutions, Sri Lanka’s information landscape remains remarkably fragmented. Every day, government ministries, regulatory authorities, courts, universities and public agencies publish thousands of documents ranging from gazettes and legislation to court judgments, policy papers, economic reports and statistical publications. While these records are technically available to the public, they are dispersed across hundreds of independent websites, stored in incompatible formats and rarely connected in any meaningful way.
For researchers, lawyers, businesses and even artificial intelligence systems, locating accurate information often means navigating dozens of separate websites, downloading lengthy PDF documents and manually piecing together information from multiple sources. The challenge has never been a shortage of data. Rather, it has been the absence of a unified infrastructure capable of connecting that information into a coherent knowledge network.
To address this long-standing challenge, Lanka Data Net (LDN) has developed the Open Public Information Network (OpenPIN), an automated backend data-harvesting tool that continuously discovers, organises and structures Sri Lanka’s publicly available information into a single interconnected digital knowledge ecosystem. While invisible to end users, OpenPIN performs one of the most technically demanding tasks within the Lanka Data platform, serving as the foundation upon which intelligent search, analytics and AI-powered services are built.
To encourage transparency, collaboration and community-driven innovation, the core components of OpenPIN have been made available as open-source projects through GitHub and Hugging Face. By releasing the underlying code and AI resources to the public, Lanka Data Net enables developers, researchers and organisations to explore, contribute to and build upon the platform, fostering a broader ecosystem for Sri Lankan data engineering, retrieval systems and sovereign AI development. This open approach supports reproducible research, accelerates innovation and reinforces OpenPIN’s vision of creating an accessible, trusted and scalable public information infrastructure for Sri Lanka.
Unlike conventional search engines that simply index webpages, OpenPIN functions as a sophisticated data engineering pipeline operating around the clock. Its role is not merely to collect documents but to transform fragmented public information into structured, machine-readable knowledge that can be understood by both humans and artificial intelligence. The platform continuously scans publicly accessible websites, government publications, gazettes, legal repositories and institutional portals, identifying new information as it becomes available and integrating it into a unified database.
One of the greatest technical challenges in building national data infrastructure lies in the inconsistency of public information. Government agencies often publish documents using different templates, naming conventions and file formats. Important records may exist as scanned PDF files, HTML pages, spreadsheets or even image-based documents that are difficult for traditional search engines to interpret. OpenPIN addresses this complexity through an automated processing pipeline that extracts meaningful content, cleans inconsistencies, standardises metadata and converts disconnected information into structured datasets suitable for intelligent retrieval and analysis.
The result is far more than a searchable archive. OpenPIN by Lanka Data Net (LDN) creates relationships between documents that would otherwise remain isolated. A single Act of Parliament can be connected with subsequent amendments, related Supreme Court judgments, gazette notifications, regulatory circulars, parliamentary debates and associated policy documents. Instead of viewing public records as individual files stored on separate websites, the platform constructs an interconnected knowledge network where information is linked according to meaning and context.
This represents a significant departure from the traditional design of government information systems. Historically, public data has been organised according to institutional boundaries, requiring users to know which ministry, department or regulatory authority originally published a particular document. Such an approach reflects the structure of government rather than the needs of citizens. OpenPIN adopts an entirely different philosophy by organising information around user intent rather than organisational hierarchy.
A business researching taxation, for example, no longer needs to understand which government agencies publish tax legislation, gazette notifications and administrative circulars. Likewise, a legal researcher examining environmental regulations can access connected information drawn from multiple institutions without manually searching each website independently. By removing these institutional barriers, OpenPIN introduces what Lanka Data describes as a citizen-centric model of digital public infrastructure, where information is connected according to practical questions and real-world use rather than administrative ownership.
Beyond improving access to information, OpenPIN by Lanka Data Net (LDN) has become a critical component in Sri Lanka’s emerging artificial intelligence ecosystem. One of the greatest weaknesses of globally trained Large Language Models is their limited understanding of local information. When asked questions about Sri Lankan legislation, taxation or regulatory matters, many international AI systems generate responses that appear convincing but are factually incorrect because they lack access to authoritative domestic sources. This phenomenon, commonly known as AI hallucination, has become one of the principal obstacles to deploying artificial intelligence in professional environments.
OpenPIN has been designed specifically to overcome this challenge. By continuously collecting, validating and structuring Sri Lanka’s public information, it provides a trusted knowledge foundation for Retrieval-Augmented Generation (RAG), a modern AI architecture that retrieves relevant documents before generating responses. Rather than relying solely on statistical predictions, AI systems built on Lanka Data’s infrastructure can reference structured local information when answering complex questions.
According to Lanka Data Net, OpenPIN has already organised more than one billion localised data points spanning legislation, business information, taxation, education, economics and public administration. This extensive knowledge base now forms the backbone of Chat2Find, Lanka Data’s AI-powered search platform, enabling responses to be grounded in current, structured information rather than relying exclusively on the underlying language model’s training data. By linking AI responses directly to authoritative public records, the platform aims to improve both factual accuracy and user confidence.
Although OpenPIN operates entirely behind the scenes, its influence extends across the broader Lanka Data ecosystem. The harvested datasets support legal research, business intelligence, regulatory analysis, taxation services, public policy research and enterprise knowledge management. Every search query, analytical report and AI-assisted response ultimately depends upon the continuous flow of structured information generated by the OpenPIN infrastructure.
Its importance also reflects a broader shift taking place internationally. Around the world, governments and technology companies are increasingly recognising that artificial intelligence cannot succeed without robust national data infrastructure. Powerful language models alone are insufficient if they cannot access reliable, up-to-date local information. As countries pursue digital sovereignty and invest in domestic AI capabilities, structured national knowledge repositories have become as strategically important as computing power itself.
Within this context, OpenPIN by Lanka Data Net (LDN) represents one of Sri Lanka’s most ambitious attempts to build foundational data infrastructure capable of supporting future digital services. Rather than focusing solely on creating another search engine or document repository, Lanka Data has concentrated on solving the far more complex problem of connecting disconnected information at scale. It is this invisible layer of continuous data collection, processing and organisation that enables intelligent applications to operate with greater accuracy and contextual understanding.
As Sri Lanka continues its digital transformation, the demand for reliable public information will only intensify. Governments will require better data sharing, businesses will seek faster regulatory intelligence, researchers will depend upon comprehensive knowledge networks and artificial intelligence will increasingly rely on trusted local datasets. Meeting these demands requires more than publishing information online; it requires building infrastructure capable of transforming isolated documents into connected knowledge.
OpenPIN illustrates how that transformation can be achieved. By continuously harvesting, cleaning and structuring Sri Lanka’s public information, it provides the invisible digital backbone that allows information to move beyond static documents and become part of a living national knowledge network. In doing so, it not only improves access to public information today but also lays the technological foundation for the next generation of AI-powered services, digital governance and data-driven innovation across Sri Lanka.
