長時間序列格點數據管理平臺的設計與實踐
作者:
作者單位:

作者簡介:

通訊作者:

中圖分類號:

基金項目:


Design and Practice of Long-Term Sequential Grid Data Management Platform
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 圖/表
  • |
  • 訪問統計
  • |
  • 參考文獻
  • |
  • 相似文獻
  • |
  • 引證文獻
  • |
  • 資源附件
  • |
  • 文章評論
    摘要:

    隨著數值模式時空分辨率的提升,數據量急劇增加,長序列數據很難直接通過文件拷貝或者網絡傳輸方式為用戶提供數據服務。為此,筆者設計實現了一種分布式管理平臺,該平臺根據用戶定制的數據需求,運用預報要素、空間范圍、時間尺度等約束條件,抽取或根據區域參數裁剪指定氣象要素,生成精簡數據進行用戶服務。該平臺集成了搜索引擎、格點數據解碼、內存數據庫技術以及分布式框架,實現跨操作系統的統一接口調用和數據快速獲取,有效解決用戶訪問長時間序列歷史資料的難題。實驗測試顯示,該平臺在格點數據管理規模和訪問效率方面均表現出色。特別是在北京2022年冬季奧運會和冬殘奧會氣象保障服務中,該平臺發揮了重要作用,展現了其實際應用的價值和潛力。

    Abstract:

    With the rapid development of numerical weather prediction services, the resolution and forecasting lead time of meteorological models have significantly improved, leading to an exponential growth in the volume of forecast data output. As a national meteorological model research and operational centre, CMA Earth System Modeling and Prediction Center (CEMC) currently produces daily gridded data outputs of 0.76 TB, with an annual output reaching 155.12 TB. Given the enormous data volumes, researchers’ preferences for data access are evolving. Wagemann predicts that future scientific users increasingly prefer cloud platforms or other interfaces for data access rather than solely relying on downloads. To address these issues, this paper proposes a lightweight distributed parallel processing framework for gridded data management, aiming to streamline data management processes and enhance data access speed. The core design philosophy revolves around leveraging search engine technology for rapid metadata retrieval and gridded data decoding techniques for efficient data acquisition. To mitigate performance penalties from repetitive decoding, the framework decodes gridded data files once and supports multiple retrievals and extractions, significantly accelerating data access. Additionally, it supports cross-platform data access, facilitating easier data acquisition for researchers. The framework adopts a three-tier architecture: the data layer stores data, the algorithm layer implements core search and cataloguing algorithms, and the business layer interfaces directly with user needs. The framework implements crucial functions such as gridded data cataloguing, extraction, and clipping. During cataloguing, users invoke the cataloguing interface and input parameters (e.g., original data file paths, index names, index types), and the system automatically parses file metadata and generates indexes. For data extraction, users call the retrieval interface with specific parameters to obtain designated data. Moreover, the framework supports precise extraction of specified latitudinal and longitudinal data segments by configuring cropping parameters. It reduces decoding time by creating indexes based on binary storage characteristics, utilises an inverted index value-id model for rapid data location retrieval, enhances processing performance through GlusterFS shared storage and Celery distributed message queues, and ensures efficient and stable data transmission using gRPC technology for C/S communication. Practical tests and applications demonstrate the framework’s exceptional performance in handling massive meteorological data. Notably, it successfully processes petabyte-scale gridded data during the Beijing Winter Olympics meteorological support services, significantly improving data access efficiency. Additionally, the framework supports flexible processing and scalable upgrades for various file formats to meet diverse user needs. By integrating advanced search engine technology, gridded data decoding methods, and a distributed cluster framework, the platform not only enables rapid data retrieval and efficient access but also satisfies researchers’ urgent demand for cross-platform data access. As meteorological data continues to grow, this platform holds significant potential to play a pivotal role in various fields, offering more robust data support for weather forecasting, scientific research, and operational applications.

    參考文獻
    相似文獻
    引證文獻
引用本文

賈曉振,胡江凱,王大鵬,梁晨.長時間序列格點數據管理平臺的設計與實踐[J].氣象科技,2024,52(6):797~806

復制
分享
文章指標
  • 點擊次數:
  • 下載次數:
  • HTML閱讀次數:
  • 引用次數:
歷史
  • 收稿日期:2023-12-12
  • 定稿日期:2024-10-09
  • 錄用日期:
  • 在線發布日期: 2024-12-25
  • 出版日期:
您是第位訪問者
技術支持:北京勤云科技發展有限公司
午夜欧美大片免费观看,欧美激情综合五月色丁香,亚洲日本在线视频观看,午夜精品福利在线
>