Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop

Li Chaokui; Zhao Yanan; Xiao Keyan; Chen Jianhui

doi:doi:10.11648/j.ajasr.20190501.12

| Peer-Reviewed |

Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop

Li Chaokui, Zhao Yanan, Xiao Keyan, Chen Jianhui

Received: 24 January 2019 Accepted: 4 March 2019 Published: 28 March 2019

Views: Downloads:

Download PDF

Share This Article

Twitter
Linked In
Facebook

Abstract

With the emergence of big data of TB and PB geological and mineral resources, the storage of large geological data has become a worldwide problem puzzling geologists. The traditional storage and service model of geological data is facing a great challenge. For example, when the scale of data increases dramatically, general relational database can not solve the problem of insufficient scalability, stability and efficiency of database system. In response to the above problems, this paper proposes a new method of geological and mineral data storage based on cloud computing environment combined with hadoop. Taking the mineral resources potential evaluation data of Chongqing as the research object, The proposed method in this paper is compared with the traditional Oracle database storage method in data storage experiments: (1) Small file optimization comparative experiment; (2) Hadoop and Oracle comparative experiment. The performance of writing operation, memory occupancy, data import and data export are tested in different way, and the comparison chart of performance is given. The experimental results show that the new storage method proposed in this paper is more efficient than the traditional method. At the same time, it effectively overcomes the problem of small file storage in Hadoop storage. The research results provide a new technical for the storage and management of geological and mineral data all over the country.

DOI	10.11648/j.ajasr.20190501.12
Published in	American Journal of Applied Scientific Research (Volume 5, Issue 1, March 2019)
Page(s)	6-16
Creative Commons	This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.
Copyright	Copyright © The Author(s), 2024. Published by Science Publishing Group

Keywords

Geological and Mineral Data, Hadoop, Oracle, Storage of Small Files

References

[1]	Zhu, Y. X. and Hou, J. G., 2014. Research on geological data storage and management based on large data// Jiangsu province surveying and mapping geographic information society Proceedings of the 2014 academic annual meeting.
[2]	Chen, J. P., Li, J., Cui, N., and Yu, P. P., 2015. The construction and application of geological cloud under the big data background. Geological Bulletin of China, 34(7):1260-1265.
[3]	Qian, X., and Zhang, D. Y., 2014. Application of geological exploration in geological prospecting. Luoyang: Journal of Henan Science and Technology, 2014(4):56-56.
[4]	Zhang, B., 2009. Based on ArcGIS Fugu county geological disaster database established and the evaluation of vulnerable areas research. Xi'an Chang'an University.
[5]	Pu, K., 2009. Design and implementation of multi-source geological spatial database storage management system. Chengdu: University of Electronic Science and Technology of China.
[6]	Liu, C. J., 2011. The research and development of the comprehensive geological database management system. Changsha: Central South University.
[7]	Zhai, Y. D., 2011. The reliability of the Hadoop distributed file system (HDFS) research and optimization. Wuhan: Huazhong University of Science and Technology.
[8]	Li, J., Chen, J. P. and Wang, X., 2015. A study of the storage technology of geological big data. Geological Bulletin of China, 2015, 34(8): 1589-1594.
[9]	Hao, S. K., 2012. Brief Analysis of the Architecture of Hadoop HDFS and MapReduce. Designing Techniques of Posts and Telecommuni, 2012(7):37-42.
[10]	Chen, K. and Zheng, W. M., 2009. Cloud computing: System instances and current research. Journal of Software, 20(5):1337-1348.
[11]	Armbrust, M., Fox, A., Griffith, R., et al. 2010. A view of cloud computing. Communications of the ACM, 53(4):50-58.
[12]	Ma, H. T., 2015. Based on the nested HBase data storage system design and implementation. Hangzhou: Zhejiang University.
[13]	Li, X. A., 2005. Future development of the database technology. Journal of Shangdong College of Electric Power, 8(2):40-43.
[14]	Cui, J. S., 2014. The Present Situation and Tendency in the Integration of Remote Sensing and Geographic Information System. TopCapital, 2014(10):247-247.
[15]	Li, C. K., Yan, W. Y. and Xiao, K. Y. 2015. The construction of geological database with MapGIS and Oracle. Geological Bulletin of China, 34(7):1359-1364.
[16]	Shvachko, K., Kuang, H. and Radia, S., 2010. The hadoop distributed filesystem. In Mass Storage Systems and Technologies (MSST), IEEE 26th Symposium. 2010:1-10.
[17]	Li, M., Cao, S. and Qin, Z. G., 2016. Storage Optimization Method of Small Files Based on Hadoop. Chengdu: Journal of University of Electronic Science and Technology of China, 2016(1):141-145.
[18]	Henzinger, T., Jhala, R., Majumdar, R., et al. 2002. Lazy Abstraction/ /POPL’02 Proc. of the 29th ACM SIGPLAN-SIGACT symposium onPrinciples of programming languages, New York, NY, USA: ACM Press.
[19]	Godefroid P., 1997. Model Checking for Programming Languages using VeriSoft//Proc. of the 24th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, New York, NY, USA: ACM Press.
[20]	Chang, F., Dean, J., Ghemawat, S., et al. 2008. Bigtable: A Distributed Storage System for Structured Data. Acm Transactions on Computer Systems, 26(2):205-218.

Cite This Article

Plain Text BibTeX RIS

APA Style

Li Chaokui, Zhao Yanan, Xiao Keyan, Chen Jianhui. (2019). Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop. American Journal of Applied Scientific Research, 5(1), 6-16. https://doi.org/10.11648/j.ajasr.20190501.12

Copy | Download

ACS Style

Li Chaokui; Zhao Yanan; Xiao Keyan; Chen Jianhui. Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop. Am. J. Appl. Sci. Res. 2019, 5(1), 6-16. doi: 10.11648/j.ajasr.20190501.12

Copy | Download

AMA Style

Li Chaokui, Zhao Yanan, Xiao Keyan, Chen Jianhui. Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop. Am J Appl Sci Res. 2019;5(1):6-16. doi: 10.11648/j.ajasr.20190501.12

Copy | Download

@article{10.11648/j.ajasr.20190501.12,
  author = {Li Chaokui and Zhao Yanan and Xiao Keyan and Chen Jianhui},
  title = {Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop},
  journal = {American Journal of Applied Scientific Research},
  volume = {5},
  number = {1},
  pages = {6-16},
  doi = {10.11648/j.ajasr.20190501.12},
  url = {https://doi.org/10.11648/j.ajasr.20190501.12},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajasr.20190501.12},
  abstract = {With the emergence of big data of TB and PB geological and mineral resources, the storage of large geological data has become a worldwide problem puzzling geologists. The traditional storage and service model of geological data is facing a great challenge. For example, when the scale of data increases dramatically, general relational database can not solve the problem of insufficient scalability, stability and efficiency of database system. In response to the above problems, this paper proposes a new method of geological and mineral data storage based on cloud computing environment combined with hadoop. Taking the mineral resources potential evaluation data of Chongqing as the research object, The proposed method in this paper is compared with the traditional Oracle database storage method in data storage experiments: (1) Small file optimization comparative experiment; (2) Hadoop and Oracle comparative experiment. The performance of writing operation, memory occupancy, data import and data export are tested in different way, and the comparison chart of performance is given. The experimental results show that the new storage method proposed in this paper is more efficient than the traditional method. At the same time, it effectively overcomes the problem of small file storage in Hadoop storage. The research results provide a new technical for the storage and management of geological and mineral data all over the country.},
 year = {2019}
}

Copy | Download

TY  - JOUR
T1  - Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop
AU  - Li Chaokui
AU  - Zhao Yanan
AU  - Xiao Keyan
AU  - Chen Jianhui
Y1  - 2019/03/28
PY  - 2019
N1  - https://doi.org/10.11648/j.ajasr.20190501.12
DO  - 10.11648/j.ajasr.20190501.12
T2  - American Journal of Applied Scientific Research
JF  - American Journal of Applied Scientific Research
JO  - American Journal of Applied Scientific Research
SP  - 6
EP  - 16
PB  - Science Publishing Group
SN  - 2471-9730
UR  - https://doi.org/10.11648/j.ajasr.20190501.12
AB  - With the emergence of big data of TB and PB geological and mineral resources, the storage of large geological data has become a worldwide problem puzzling geologists. The traditional storage and service model of geological data is facing a great challenge. For example, when the scale of data increases dramatically, general relational database can not solve the problem of insufficient scalability, stability and efficiency of database system. In response to the above problems, this paper proposes a new method of geological and mineral data storage based on cloud computing environment combined with hadoop. Taking the mineral resources potential evaluation data of Chongqing as the research object, The proposed method in this paper is compared with the traditional Oracle database storage method in data storage experiments: (1) Small file optimization comparative experiment; (2) Hadoop and Oracle comparative experiment. The performance of writing operation, memory occupancy, data import and data export are tested in different way, and the comparison chart of performance is given. The experimental results show that the new storage method proposed in this paper is more efficient than the traditional method. At the same time, it effectively overcomes the problem of small file storage in Hadoop storage. The research results provide a new technical for the storage and management of geological and mineral data all over the country.
VL  - 5
IS  - 1
ER  -

Copy | Download

Author Information

Li Chaokui

National-Local Joint Engineering Laboratory of Geospatial Information Technology, Hunan University of Science and Technology, Xiangtan, China
Zhao Yanan

National-Local Joint Engineering Laboratory of Geospatial Information Technology, Hunan University of Science and Technology, Xiangtan, China
Xiao Keyan

Institute of Mineral Resources, Chinese Academy of Geological Sciences, Beijing, China
Chen Jianhui

National-Local Joint Engineering Laboratory of Geospatial Information Technology, Hunan University of Science and Technology, Xiangtan, China

Download PDF

Sections

Plain Text BibTeX RIS

APA Style

Li Chaokui, Zhao Yanan, Xiao Keyan, Chen Jianhui. (2019). Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop. American Journal of Applied Scientific Research, 5(1), 6-16. https://doi.org/10.11648/j.ajasr.20190501.12

Copy | Download

ACS Style

Li Chaokui; Zhao Yanan; Xiao Keyan; Chen Jianhui. Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop. Am. J. Appl. Sci. Res. 2019, 5(1), 6-16. doi: 10.11648/j.ajasr.20190501.12

Copy | Download

AMA Style

Li Chaokui, Zhao Yanan, Xiao Keyan, Chen Jianhui. Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop. Am J Appl Sci Res. 2019;5(1):6-16. doi: 10.11648/j.ajasr.20190501.12

Copy | Download

@article{10.11648/j.ajasr.20190501.12,
  author = {Li Chaokui and Zhao Yanan and Xiao Keyan and Chen Jianhui},
  title = {Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop},
  journal = {American Journal of Applied Scientific Research},
  volume = {5},
  number = {1},
  pages = {6-16},
  doi = {10.11648/j.ajasr.20190501.12},
  url = {https://doi.org/10.11648/j.ajasr.20190501.12},
  eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajasr.20190501.12},
  abstract = {With the emergence of big data of TB and PB geological and mineral resources, the storage of large geological data has become a worldwide problem puzzling geologists. The traditional storage and service model of geological data is facing a great challenge. For example, when the scale of data increases dramatically, general relational database can not solve the problem of insufficient scalability, stability and efficiency of database system. In response to the above problems, this paper proposes a new method of geological and mineral data storage based on cloud computing environment combined with hadoop. Taking the mineral resources potential evaluation data of Chongqing as the research object, The proposed method in this paper is compared with the traditional Oracle database storage method in data storage experiments: (1) Small file optimization comparative experiment; (2) Hadoop and Oracle comparative experiment. The performance of writing operation, memory occupancy, data import and data export are tested in different way, and the comparison chart of performance is given. The experimental results show that the new storage method proposed in this paper is more efficient than the traditional method. At the same time, it effectively overcomes the problem of small file storage in Hadoop storage. The research results provide a new technical for the storage and management of geological and mineral data all over the country.},
 year = {2019}
}

Copy | Download

TY  - JOUR
T1  - Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop
AU  - Li Chaokui
AU  - Zhao Yanan
AU  - Xiao Keyan
AU  - Chen Jianhui
Y1  - 2019/03/28
PY  - 2019
N1  - https://doi.org/10.11648/j.ajasr.20190501.12
DO  - 10.11648/j.ajasr.20190501.12
T2  - American Journal of Applied Scientific Research
JF  - American Journal of Applied Scientific Research
JO  - American Journal of Applied Scientific Research
SP  - 6
EP  - 16
PB  - Science Publishing Group
SN  - 2471-9730
UR  - https://doi.org/10.11648/j.ajasr.20190501.12
AB  - With the emergence of big data of TB and PB geological and mineral resources, the storage of large geological data has become a worldwide problem puzzling geologists. The traditional storage and service model of geological data is facing a great challenge. For example, when the scale of data increases dramatically, general relational database can not solve the problem of insufficient scalability, stability and efficiency of database system. In response to the above problems, this paper proposes a new method of geological and mineral data storage based on cloud computing environment combined with hadoop. Taking the mineral resources potential evaluation data of Chongqing as the research object, The proposed method in this paper is compared with the traditional Oracle database storage method in data storage experiments: (1) Small file optimization comparative experiment; (2) Hadoop and Oracle comparative experiment. The performance of writing operation, memory occupancy, data import and data export are tested in different way, and the comparison chart of performance is given. The experimental results show that the new storage method proposed in this paper is more efficient than the traditional method. At the same time, it effectively overcomes the problem of small file storage in Hadoop storage. The research results provide a new technical for the storage and management of geological and mineral data all over the country.
VL  - 5
IS  - 1
ER  -

Copy | Download