519502.prezentacija_webmining

8/2/2019 519502.prezentacija_webmining

1/107

Mining Social and Semantic Network Data on the Web

Markus Schatten, PhD

University of ZagrebFaculty of Organization and Informatics

May 4, 2011


2/107

Introduction

Web 2.0, Semantic Web, Web 3.0

Network science

Data collection

Semistructured data

XPath

Regular expressionsAjax

APIs

Data analysis

Network matricesFolksonomy model

Logic programming

Case study - CROSBI

2 of 54


3/107

Outline

Introduction


Network science

Data collection

Data analysis

Case study - CROSBI

3 of 54


4/107

Questions

Why mining social and semantic network data?

How to collect these data? What are the common obstacles and how to solve them?

How to analyze these data?

4 of 54


5/107

Outline

Introduction


Network science

Data collection

Data analysis

Case study - CROSBI

5 of 54


6/107


Why should one bother with these buzzwords?

6 of 54


7/107

Why mining (social) web data?

Millions of people,

7 of 54


8/107


Millions of people, on thousands of sites,

7 of 54


9/107


Millions of people, on thousands of sites, across hundreds ofdiverse cultures,

7 of 54


10/107


Millions of people, on thousands of sites, across hundreds ofdiverse cultures, leave trails

7 of 54


11/107


Millions of people, on thousands of sites, across hundreds ofdiverse cultures, leave trails about various aspects of their lives...

7 of 54


12/107



Social systems are systems of communication

(Niklas Luhman)

7 of 54


13/107




(Niklas Luhman)

By analyzing the trails (the results of social systems structural

coupling) we can analyze the social system itself:

7 of 54


14/107




(Niklas Luhman)

By analyzing the trails (the results of social systems structural

coupling) we can analyze the social system itself:politics, culture,

media, business, technology, science ...

7 of 54


15/107

Outline

Introduction


Network science

Data collection

Data analysis

Case study - CROSBI

8 of 54


16/107

The new science of networks

source: The New York Times (April 3rd, 1933., p. 17).

source: An Attraction Network in a Fourth Grade Class (Moreno, Who shall survive?, 1934).

9 of 54


17/107

Political and financial networks

Mark Lombardi (1980s & 1990s)

10 of 54


18/107

Terrorist networks

11 of 54


19/107

Board of directors membership networks

source: http://theyrule.net

12 of 54
http://theyrule.net/http://theyrule.net/


20/107

On-line social networks

13 of 54


21/107

Internet

source: Bill Cheswick http://www.cheswick.com/ches/map/gallery/index.html

14 of 54
http://www.cheswick.com/ches/map/gallery/index.htmlhttp://www.cheswick.com/ches/map/gallery/index.html


22/107

Air traffic networks

source: Northwest Airlines WorldTraveler Magazine

15 of 54


23/107

Railway networks

source: TRTA, March 2003 - Tokyo rail map

16 of 54


24/107

Semantic networks

source: http://wordnet.princeton.edu/man/wnlicens.7WN

17 of 54
http://wordnet.princeton.edu/man/wnlicens.7WNhttp://wordnet.princeton.edu/man/wnlicens.7WN


25/107

Gene networks

source: http://www.zaik.uni-koeln.de/bioinformatik/regulatorynets.html.en

18 of 54
http://www.zaik.uni-koeln.de/bioinformatik/regulatorynets.html.enhttp://www.zaik.uni-koeln.de/bioinformatik/regulatorynets.html.en


26/107

Food chain networks

source: http://marinebio.org/Oceans/Biotic-Structure.asp19 of 54
http://marinebio.org/Oceans/Biotic-Structure.asphttp://marinebio.org/Oceans/Biotic-Structure.asp


27/107

On-line social and semantic networks

Where can we find networks? social networking sites

20 of 54


28/107



forums, buletinboards, discussion groups, mailing lists

20 of 54


29/107




blogs, microblogs, vlogs

20 of 54


30/107





wikis, semantic wikis

20 of 54


31/107





wikis, semantic wikis social tagging

20 of 54


32/107






RSS feeds, RDF repositories, metadata collections

20 of 54


33/107






RSS feeds, RDF repositories, metadata collections...

20 of 54


34/107







A social network can be found on any application where users

communicate.

20 of 54


35/107







A social network can be found on any application where users

communicate.A semantic network can be found in any specification where

concepts are meaningfully connected.

20 of 54


36/107

Outline

Introduction


Network science

Data collectionSemistructured dataXPathRegular expressionsAjaxAPIs

Data analysis

Case study - CROSBI21 of 54


37/107

Common obstacles in data collection

Semi-structured data (HTML,

XHTML, XML, RDF, OWL ...)

22 of 54


38/107




The need for pattern matching

22 of 54


39/107




The need for pattern matching

Client-side content generation

22 of 54


40/107

Semistructured data and XPath

query language for semistructured data

part of the W3C standard

uses path expressions to navigate through documents

has a standard library with over 100 useful functions

Example:

/html/body/div[@id=background]/div/div[@id=mainBar]/div/p[last()-1]

23 of 54


41/107

XPather (Firefox add-on)

24 of 54


42/107

Regular expressions

Useful for extracting various formatted data and searching for

keywordsExample (date matching)

27-9-2001 [0-9]{1,2}[0-9]{1,2}[1-2][0-9]{3}

25 of 54


43/107

Client side generated content

Some sites use Ajax to generate data on page load or user

interaction (e.g. mouse click, mouse over etc.)

26 of 54


44/107



interaction (e.g. mouse click, mouse over etc.) In combination with obfuscated JavaScript code this is hard to

scrape!

Current solutions:

Browser scripting (can be slow and cumbersome) - e.g. withselenium

26 of 54


45/107




scrape!

Current solutions:

Browser scripting (can be slow and cumbersome) - e.g. withselenium

Headless browser - e.g. mechanize, spidermonkey, HtmlUnit

26 of 54


46/107




scrape!

Current solutions:

Browser scripting (can be slow and cumbersome) - e.g. withselenium Headless browser - e.g. mechanize, spidermonkey, HtmlUnit

26 of 54


47/107

Open APIs

Most popular social web sites have an open API that allow you

to connect your application to their site (Facebook, Twitter,

Wikipedia, ...)

27 of 54

O API


48/107

Open APIs



Wikipedia, ...)

Still such APIs often have drawbacks for network datacollection:

27 of 54

O API


49/107

Open APIs



Wikipedia, ...)

Still such APIs often have drawbacks for network datacollection: Limited access rights (e.g. Facebook)

27 of 54

O API


50/107

Open APIs



Wikipedia, ...)

Still such APIs often have drawbacks for network datacollection: Limited access rights (e.g. Facebook) Limited queries per time interval (e.g. Twitter) etc.

27 of 54

O tli


51/107

Outline

Introduction


Network science

Data collection

Data analysisNetwork matrices

Folksonomy modelLogic programming

Case study - CROSBI

28 of 54


52/107

Networks are most often analyzed using graph theory:

DefinitionA graphG is the pair(N, E) wherebyN represents the set of verticles ornodes, andE N N the set of edges connecting pairs fromN.

29 of 54


53/107



DefinitionA directed graph or digraphG is the pair(N, A), wherebyN represents

the set of nodes, andA N N the set of ordered pairs of elementsfromN that represent the set of graph arcs.

29 of 54


54/107



DefinitionA directed graph or digraphG is the pair(N, A), wherebyN represents

the set of nodes, andA N N the set of ordered pairs of elementsfromN that represent the set of graph arcs.

DefinitionA valued or weighted graph GV is the triple(N, A, V) wherebyNrepresents the set of nodes or verticles, A N N the set of pairs of

elements fromN that represent the set of graph arcs, and V : N R afunction that attaches values or weights to nodes.

29 of 54


55/107

DefinitionA bipartite graphG = (N, E) is a graph for which set of nodes thereexists an partitionN = {X, Y}, such that every arc has one end in X, aand the other in Y .

30 of 54


56/107

DefinitionA bipartite graphG = (N, E) is a graph for which set of nodes thereexists an partitionN = {X, Y}, such that every arc has one end in X, aand the other in Y .

DefinitionA multigraphG = (N, E) is a graph in whichE is a multiset, e.g. therecan exist multiple edges between two nodes.

30 of 54

Matrix representation


57/107

Matrix representation

DefinitionLetG be a graph defined with the set of nodes {n1, n2, ..., nm} and edges{e1, e2, ..., el}. For every i,j (1 i m and1 j m) we define

aij =

1, if there is an edge between nodes ni and nj

0, otherwise

Matrix A = [aij] is then the adjacency matrix of graphG. The matrix isymmetric since if there is an edge between nodes ni and nj then clearly

there is also an edge between nj and ni. Thus A = [aij] = [aji].

31 of 54

Simple graph


58/107

Simple graph

32 of 54

Digraph


59/107

Digraph

33 of 54

Digraph


60/107

Digraph

34 of 54

Weighted graph


61/107

Weighted graph

35 of 54

Bipartite graph


62/107

Bipartite graph

36 of 54

Multigraph


63/107

Multigraph

37 of 54

Matrix algebra and interpretation


64/107

g p

Matrices can be transposed, multiplied ...

38 of 54



65/107

g p


Examples:P - process flow matrix (p p, p - number of processes)

38 of 54



66/107

g p


Examples:P - process flow matrix (p p, p - number of processes)W - co-worker matrix (w w, w - number of workers)

38 of 54



67/107

g p


Examples:P - process flow matrix (p p, p - number of processes)W - co-worker matrix (w w, w - number of workers)O - process responsibility matrix (p z)

38 of 54



68/107


Examples:P - process flow matrix (p p, p - number of processes)W - co-worker matrix (w w, w - number of workers)O - process responsibility matrix (p z)WW - co-workers of co-workers

38 of 54



69/107



PT - inversed process flow

38 of 54



70/107




OTO - matrix of workers who are responsible for the same

processes

38 of 54



71/107





processes

PO - matrix of workers who are responsible for the forthcoming

processes

38 of 54



72/107





processes


processesPTO - matrix of workers who are responsible for the previous

processes

38 of 54



73/107





processes


processesPTO - matrix of workers who are responsible for the previous

processes

...

38 of 54

ACI model (Mika 2007)


74/107

A - actors

39 of 54



75/107

A - actors C - concepts

39 of 54



76/107


I - instances

39 of 54



77/107


I - instances

T A C I - folksonomy

39 of 54



78/107


I - instances

T A C I - folksonomyA tripartite hypergraph which can be reduced into three bipartite

graphs:

39 of 54



79/107


I - instances


graphs:

AC - network of actors and concepts

39 of 54



80/107


I - instances


graphs:


AI - network of actors and instances

39 of 54



81/107


I - instances


graphs:


AI - network of actors and instances CI - network of concepts and instances

39 of 54

Delicious - analysis


82/107

40 of 54

Problems with matrix representation


83/107

41 of 54

Problems with matrix representation


84/107

Only the 1-s contain information

41 of 54

Sparse matrix


85/107

42 of 54

Problems with sparse matrix


86/107

If A is connected to B, then B is connected to A (undirected

graphs)

43 of 54



87/107


graphs)

How to find friends of friends?

43 of 54



88/107


graphs)

How to find friends of friends? Is there a path from node A to node B? Which path is that?

43 of 54



89/107


graphs)


How to compute the previously outlined graphs?

43 of 54



90/107


graphs)


How to compute the previously outlined graphs?

43 of 54

Logic programming as a solution


91/107

Deductive languages like Prolog, FLORA-2

44 of 54



92/107

Deductive languages like Prolog, FLORA-2 Advantages:

Declarative problem solving (on a higher level) Very appropriate for expressing mathematical structures

44 of 54



93/107

Deductive languages like Prolog, FLORA-2 Advantages:

Declarative problem solving (on a higher level) Very appropriate for expressing mathematical structures

Disadvantages: Combinatoric explosion (use heuristics and predicate tabling)

44 of 54

Examples


94/107

If A is connected to B, then B is connected to A

link( A, B ) :- link( B, A ).

45 of 54

Examples


95/107

Friend of a friend

ff( A, B ) :- link( A, C ), link( C, B ).

46 of 54

Examples


96/107

Is there a path from node A to node B?

path( A, B ) :- link( A, B ).path( A, B ) :- link( A, C ), path( C, B ).

47 of 54

Examples


97/107

Which path is that?

whichpath( X, Y, [ X, Y ] ) :- link( X, Y ).

whichpath( X, Y, [ X, Z | T ] ) :-

link( X, Z ),

whichpath( Z, Y, [ Z | T ] ).

48 of 54

Outline


98/107

Introduction


Network science

Data collection

Data analysis

Case study - CROSBI

49 of 54

Project outline


99/107

Croatian Scientific Bibliography (http://bib.irb.hr)

50 of 54

Project outline
http://bib.irb.hr/http://bib.irb.hr/


100/107

Croatian Scientific Bibliography (http://bib.irb.hr) Scientists enter data about their research activities by them

selves which makes it a social application

50 of 54

Project outline


101/107



Research hypotheses:

50 of 54

Project outline


102/107




1. analyze two social systems (the Croatian scientific community and

the Croatian public) and see how the most important researchconcepts (as viewed by the scientific community) are interpretedby the public (does the public understand most important conceptsfrom Croatian science in the last decade?).

50 of 54

Project outline


103/107




1. analyze two social systems (the Croatian scientific community and

the Croatian public) and see how the most important researchconcepts (as viewed by the scientific community) are interpretedby the public (does the public understand most important conceptsfrom Croatian science in the last decade?).

2. analyze how various scientific fields are conceptually and socially

interrelated (do scientists do interdisciplinary research together ifthey have conceptually connected fields?)

50 of 54

Methodology


104/107

Social systems are represented by their trails: CROSBI as the

Croatian scientific community, Croatian Wikipedia as the

Croatian public

One system can interpret concepts from another if the concept

exists within its conceptual network (e.g. keyword or wiki page)

51 of 54

Data collection


105/107

CROSBI data has been collected using Scrapy in November

2010 (a total of 285,234 entries has been collected)

Wikipedia data has been collected using the Wikipedia API +Scrapy for parsing

52 of 54

Results


106/107

53 of 54

Bibliography


107/107

1. Adamic, L.: Why networks are interesting to study, University of Michigan, School of Information,https://open.umich.edu/education/si/si508/fall2008

2. Barratm M., Barthlemy, M., Vespignani, A.: Dynamical Processes on Complex Networks, Cambridge UniversityPress, 2008.

3. Carley, K.M.: Dynamic Network Analysis, Summary of the NRC workshop on social network modeling andanalysis, Committee on Human Factors, National Research Council, 133145, Eds: Breiger, R., Carley, K.M., andPattison, P., 2003.

4. Divjak, B., Lovrencic, A.: Diskretna matematika s teorijom grafova, TIVA & FOI, 2005

5. Krackhardt, David, and Carley, Kathleen M.: A PCANS Model of Structure in Organization, Proceedings of the1998 International Symposium on Command and Control Research and Technology, Evidence Based Research,Vienna, VA, June 1998

6. Mika, Peter: Ontologies are us: A unified model of social networks and semantics, Web Semantics: Science,Services and Agents on the World Wide Web 5(1), volume 5, 515, March 2007

7. Newman, M., Barabsi, A.-L., Watts, D. j.: The Structure and Dynamics of Networks, Princeton University Press,2006.

8. Various web sites (images, graphs)

54 of 54
https://open.umich.edu/education/si/si508/fall2008https://open.umich.edu/education/si/si508/fall2008

519502.prezentacija_webmining

Documents

Transcript of 519502.prezentacija_webmining