This project is a data collection system of the State Grid in order to analyze the financial index and stock quotation information. The system is divided into two subsystems: crawler core system and data management system. Background management personnel through the management page according to the specified target page document structure of the crawl rules and crawl task, start the task on time from the specified website to crawl the stock index, and input into the database
In the past, I paid more attention to development of software. This time I have to consider the requirements analysis, technical selection and difficult public relations, and assign the corresponding work according to the ability of each member of the team. The pressure and workload are much higher than before. I learned to adapted to it gradually, and controlled the project schedule, taught new people learning methods instead of learning knowledge patiently. My communication and management skills have been greatly improved in this project.
