Abstract:

Social network data acquisition is a vital technology and precondition for public opinion analysis. However, shielding measures of the current major social network platforms are increasingly complexity to third-party crawlers. Traditional data acquisition means are facing increasingly severe challenges. This paper proposes a data acquisition technology based on browser test components, which avoids some limitations that social networks have set up to traditional network crawlers by simulating normal users’ behaviors, in order to achieve efficiently data acquisition. The system acquires QQ group real-time message information and non real-time information. For real-time message, the result of comparative experiment shows that the overall rate of instant information collection reaches 99%, and the accuracy rate reaches 100%. For non-real-time information, the system acquires the information of group announcement, the list of group members, the group shared file and album. By contrast of data sampling, the result shows that overall rate and accuracy rate all reach 100%. The experiment proves that social network data acquisition technology based on browser test components has its advantages on data acquisition.

Key words: social network, data acquisition, user behavior simulation, browser test components