Internet Economics כלכלת האינטרנט Class 9 – predictions and scoring rules 1.
-
date post
19-Dec-2015 -
Category
Documents
-
view
216 -
download
1
Transcript of Internet Economics כלכלת האינטרנט Class 9 – predictions and scoring rules 1.
Internet Economicsכלכלת האינטרנט
Class 9 – predictions and scoring rules
1
Today
2
• Prediction markets and scoring rules.
• Some guidelines for the 2nd semester
Wisdom of the crowds
3
• The Internet is used to aggregate information from the general population.
• Creation of “information markets”.
• Main use: calculating probabilities of one-time events– Winners in elections.– Outcomes of sports events.– Political changes.– Financial events (future exchange rates, future stock values)
• In large companies: elicit information from employees.– Probability that our new smartphone will gain more than 10% of the
market.
Contracts
5
• We need to sign a contract with the market participants.
• Contract should be based on verifiable data.– Sometimes it is tricky….
– Israel will strike Iran by December 31st 2011.– Robert De-Niro will win the 2011 academy
awards for best actor in main role.
6
Winner-takes-all contract
7
• A bond is issued:– If event A occurs, it pays $100.– Otherwise, pays 0.
• For example: Saddam securityPaid out $100 if Saddam is no longer in power by June 2003.– Traded on website tradesports.com– In the hour after President Bush's war "declaration", the Saddam-gone-by-
March bet shot up nearly 10 percentage points (odds went from about 60 to 69 percent)
– Price on Jan 1, 2003: $55– Price on March 1, 2003: $70
Trading Contracts
8
• How are contracts traded?– In most prediction markets, the mechanism used
is the continuous double auction.– This is the mechanism used on the NYSE.
Winner-takes-all Contracts
9
• US 2008 elections – price history (from Iowa Electronic Market)
Scoring rules
10
• We sometime aim to more delicate contracts
• For example, think about the following simple scenario:
we do not want to aggregate the wisdom of the crowds, but the wisdom of one individual.
Example: Weatherman
11
• Tomorrow can be either sunny or rainy.• We would like to know the probability of a sunny day.• A weather forecaster knows, but he can lie.• Can we incentivize him to tell us the truth?
75% for a sunny day tomorrow
Contracts
12
• We need to sign a contract with the weatherman.
• Contract with the weatherman can depend on:– The probability declared by the forecaster.– The realization of the outcome (sunny/rainy)
• Contract cannot depend on the true probability– It is unobservable, even in retrospect.
Proper scoring rules
13
• We would like that given the contract, a utility-maximizing forecaster will report the true probability.– Via a bonus system.
• Contracts where the forecaster strictly prefers to report the truth are called proper scoring rules.– It is easy to construct scoring rules that are not proper.
Model
14
• Consider a binary outcome: z=0 or z=1– Example: z=1 means rain, z=0 means sun.
• p is the true probability of z=1.– Pr(z=1)=p, Pr(z=0)=1-p
• q is the report of the forecaster.
• The contract:– After realizing the outcome z, the forecaster pays
R(z,q).
• Are there proper scoring rules in this model?
First try: pay your prediction
15
• Consider the following contract:– R(z,q) = q
• Pay the probability that you gave to the realized outcome.
– i.e., when z=0 forecaster is paid R(0,q) = 1-q when z=1 forecaster is paid R(1,q) = q
• Expected surplus for forecaster:– p*q + (1-p)*(1-q) = (1-p)+(2p-1)q
• Forecaster will optimize q.– Report 1 if p>1/2, report 0 otherwise.
• Not a proper scoring rule.
Example: quadratic scoring rules
16
• Consider the following contract:– R(z,q) = 1 - (z-q)2 – i.e., when z=0 forecaster is paid R(0,q) = 1-q2
when z=1 forecaster is paid R(1,q) = 1-(1-q)2 =q(2-q)
• Expected surplus for forecaster:– p*R(1,q) + (1-p)*R(0,q) = pq(2-q)+(1-p)(1-q2)
• Forecaster will optimize q.– FOC:
• p(2-2q)+(1-p)(-2q) = 2p-2pq-2q+2pq = 0 q=p
• A proper scoring rule (“quadratic scoring rule”)
Example: logarithmic scoring rule
17
• Consider the following contract:– R(z,q) = zlogq + (1-z)log(1-q)– i.e., when z=0 forecaster is paid R(0,q) = log(1-q)
when z=1 forecaster is paid R(1,q) = logq
• Expected surplus for forecaster:– p*R(1,q)+(1-p)*R(0,q) = p* log(q) + (1-p)* log(1-q)
• FOC:– p(1/q)-(1-p)(1/(1-q))=0 q=p
• A proper scoring rule (“logarithmic scoring rule”)
Finding proper scoring rule
18
A general result:• n possible events (1,…,n)• Probability estimate r=r1,…,rn
• Scoring rule for each event si(ri) (i=1,…,n)• Expected score function:
Theorem: (McCarthy ‘56, Hendrickson and Buehlera'71 )
Every homogeneous and convex expected score function defines a proper scoring rule.
• Example: quadratic scoring rule H(q)=q2(2-q)+(1-q)(1-q2)
n
iiiin rsrrrH
11 )(),...,(
Summary
19
• Often one would like to estimate probabilities of future events– Aggregate information from the crowds
• Used in Information/prediction markets
• Strong ties with financial securities trade– Options, etc.
• Some beautiful economic theory behind it.
Today
20
• Prediction markets and scoring rules.
• Some guidelines for the 2nd semester
Grade• 30% presentation• 58% seminar paper• 12% participation
– You will get 12 points if you attend at least 80% of the classes in both the 1st and the 2nd semester.
– If you miss the 80% in one of the semesters, you get 0 points.– You will not get a grade in the course if you attend less than 50% of
the classes in at least one of the semesters.
• 12% problem set (“magen”)
Course duties: presentation
22
• In pairs.– Let the better speaker talk, but two speakers possible.
• Practice– Remember: presentations in real-life are much faster
than the practice talks.
• 25 minute talk. (~3 in each meeting)– Schedule will be posted in the beginning of the semester break.– You should be ready to present in the first week of the semester.
• Make it interesting to the class!– Speak to class, not to instructor.
• Language: Slides in English (preferred, not mandatory), Speak Hebrew.
Course duties: presentation
23
• Goal: present the paper to the class.
– Present what is written in the paper, no need for criticism or your interpretation.
– Give the background, and make the main results clear(do not present all results).
– If suitable, spend at least 5-10 minutes on technical details (proof idea, analysis methods etc.)
– Survey related work very briefly, only if needed. (sometimes it is good to read some of these…)
– Don’t follow the structure of the paper exactly.
– Explain in your own words.
Course duties: presentation
24
• Presentation grading will be based on:– Clarity.– Understanding.– Identifying and understanding the main results.– Quality of presentation.– Conveying complicated ideas in a simple way.– Presenting your own understanding (not just cut
and paste).
“Make everything as simple as possible, but not simpler. ”
Albert Einstein
Some presentation tips.
25
• Use large fonts – in general, rarely use fonts below 24.
• Don’t have dense slides – split them.• A figure is worth a thousand words:
If x is an individually rational revenue maximizing auction then there exists a Pure Nash equilibrium in the game.
• Of course: Be politically correct– No offensive language, images, examples etc.
X is individually rational revenue
maximizing auction
Pure Nash equilibrium exists
Some presentation tips.
26
• Use colors.• Use images.• Use animation
– But with care, do not overuse itplease
• Make it interesting, but remember that this is an academic duty. Content is the main thing. – Most effort should go on understanding the paper you
read.
Some presentation tips.
27
• Basic presentation structure:– Motivation (usually, real life problem, informal) +
Background– Informal results (if possible)– Model– Results– Technical details– Summary
• Of course, you should fit the structure to your paper.
Course duties: Problem set
28
• Optional (Magen 12%)
• Strict deadline: February 1st
• By your own. (not in pairs.)
• Available tomorrow on the website.
Course duties: Seminar paper
29
• Should be on the same topic as your presentation– With the same partner.
• Up to 10 pages, font 12, 1.5 spacing. (strict!)
• Submission time: May 30th.
Course duties: Seminar paper
30
In the paper: Part 1: Summarize the paper.
Background, main results, main technical details .– Some proofs (outline if needed), methods of data
analysis. (Tables with data should not be part of the 10 pages you submit,
can be in an appendix.)– Go to depth regarding the main result/s.
Part 2: A new research direction, – Phrase exact questions and challenges, one (small) step
toward to solution.– Idea may be theoretical, empirical or other. You need to
define the framework of the research (model, goals, questions, difficulties, data resources, suggested theorems)
Course duties: Seminar paper
31
Originality:
• Cite all materials that you use.
• Statements that are not from the paper you read should be given with a bibliographic reference.
• Especially, let me know if you intend writing any project for another course on a similar subject.
• Do not copy text, even from the paper you read.– Explain in your own language. Give new examples. Give
the intuition behind the proofs.
Timeline
32
Winter vacation starts
Winter vacation ends
Summervacation starts
All seminar topics are chosenand approved(January 1st) Students
presentations start(February 16thth)
All seminar papers are submitted(May 30th)
Problem set due date(Feb. 1st)
Presentation outline + hard scheduling constraints sent to Avi.(January 12th)
presentations
Presentation schedule posted(January 20th)
33
Good luck.
Don’t hesitate to come to me for questions or advise.