基于流程控制的國際氣象通信數據收集管理系統的設計與實現
作者:
作者單位:

作者簡介:

通訊作者:

中圖分類號:

基金項目:


Design and Implementation of International Meteorological Communication Data Collection and Management System
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 圖/表
  • |
  • 訪問統計
  • |
  • 參考文獻
  • |
  • 相似文獻
  • |
  • 引證文獻
  • |
  • 資源附件
  • |
  • 文章評論
    摘要:

    近年來,天氣和氣候業務對國外數據種類和時效要求不斷提高,為解決大量數據的收集分發壓力以及管理問題,設計并實現了基于流程控制的國際氣象通信數據收集管理系統。系統應用數據收集器Logstash、Inotify監聽機制與分析平臺Kibana等技術,支持多種通信協議,擴展Scrapy網絡爬蟲功能,通過流程控制、任務調度等方式實現大量數據快速高效收集和集成管理,有效提高數據并行處理能力和可擴展性,且設計了全流程監視,保障數據時效。系統已經上線提供服務,為業務/科研用戶提供全球觀測數據、國外衛星數據和數值預報產品等300余種,日收集數據量約2 TB。

    Abstract:

    In recent years, meteorological and climate services have increased their requirements for foreign data types and timeliness. An international meteorological communication data collection and management system based on process control is designed and implemented to solve the collection and distribution pressure and management problems of a large amount of data. The system applies the technology of Logstash, Inotify and Kibana, supports a variety of communication protocols, expands the network crawler function of Scrapy, and achieves the fast and efficient collection and integration management of a large amount of data through process control and task scheduling, effectively improving the data parallel processing ability and scalability. And the whole process of monitoring is designed to ensure data timeliness. The system has been online to provide services, providing operational/scientific research users with more than 300 kinds of global observation data, foreign satellite data and numerical prediction products; the daily data collection is about 2 TB.

    參考文獻
    相似文獻
    引證文獻
引用本文

顧文靜,王鵬,戴晴,韓鑫強.基于流程控制的國際氣象通信數據收集管理系統的設計與實現[J].氣象科技,2023,51(1):31~39

復制
分享
文章指標
  • 點擊次數:
  • 下載次數:
  • HTML閱讀次數:
  • 引用次數:
歷史
  • 收稿日期:2021-11-08
  • 定稿日期:2022-10-26
  • 錄用日期:
  • 在線發布日期: 2023-03-03
  • 出版日期:
您是第位訪問者
技術支持:北京勤云科技發展有限公司
午夜欧美大片免费观看,欧美激情综合五月色丁香,亚洲日本在线视频观看,午夜精品福利在线
>