Saturday, November 3, 2012

Skill Scores: Re: Nick Rowe

 But if both forecasters are imperfect, how do we use the data to tell us which forecaster was better?  Or how good each one is on a scale with pure guess at one end and a crystal ball at the other end? How can we estimate the degree to which a forecaster was over-confident or under-confident in his own forecasting ability? What about bias?

Nick's question had to do with probabilistic forecasts. This problem is this isn't actually covered in any econometrics or probability classes.  You have to go to a meteorology department to figure out how to do this without making stupid mistakes.


A good example of why good skill scores are needed is the following model. This model was a real model back in the day and actually used by a real weather forecaster.  It predicts on every given day that there will not be a tornado in a given town.  The forecaster claimed that he would be 98% right, but what we care about is that day when there in fact is a tornado.  He got no false positives, but way too many (100%) false negatives.  His score was "biased." (This is very different from what econometricians call bias.)

The Heidke skill score is a better predictor.

Here's a really simple example for yes-no answers:

a = Forecast = Yes, Reality = Yes.
b = Forecast = Yes, Reality = No.

c = Forecast = No, Reality = Yes
d = Forecast = No, Reality = Yes


We come up with the Heike Skill score by comparing how our model does compared to a coin flip. And how good a perfect model is compared to a coin flip. Then to make the numbers nice we take the ratio of those two results:

HSS = (number correct - expected number correct with a coin flip)/(perfect model's number - number correct with a coin flip)

This simplifies to:

$$HSS = \frac{2(ad - bc)} {(a+c)(c+d) + (a+d)(b+d)}$$ using the above definitions.

HSS of one means a perfect forecaster, a zero means the forecaster has no skill, and a negative value says that flipping a coin is actually better than the forecaster.

There are many other types of skill scores.  They differ based on how they treat rare events, non-events, and systemic vs random errors.  You can extend skill scores from a 2x2 table to a larger table for more complex forecasts. This won't do for probabilistic forecasts, however.

For probabilistic forecasts, instead of weighing false positives vs false negatives, you are weighing sharpness vs reliability.  Here is a skill score for probabilistic forecasts:


The Ignorance Skill Score

Let f be the predicted probability of an event occurring lying on the open interval (0,1).  (The ignorance skill score assumes that we are never 100% sure about anything.) Also the ignorance skill score has units of "bits."  Yes, it's the same thing we talk about when we speak of "bits" in a computer.  It traces its foundations to information theory.

And let:
$$Ignorance_t(f_t) = -log_2(f_t)$$ when the event happens at time period t,
$$Ignorance_t(f_t) = -log_2(1 - f_t)$$ when the event does not happen at time period t, and
T = the number of time periods t.

The expected ignorance is computed the normal way:
$$Ignorance(f)=\frac 1 {T}\sum_t I_t(f_t)$$

Standard errors for our estimate of ignorance are also computed the normal way.

So, back to your original question, "how do we compare probabalistic forecasters."  We can compare the ignorance of the two forecasters by seeing which one is more ignorant.


Here is a more intuitive way to understand it:

Let's define a function that is "a measure of the information content associated with the outcome of a random variable."

Since it's a measure of information, then it should have the following properties:

1) This measure of event $$A_i$$ happening depends only on the probability, $$p_i$$ of $$A_i$$ happening.

2) It's a strictly decreasing function of $$p_i$$.  This is so that the higher the probability, the less useful our prediction of event $$A_i$$.

3) It's a continuous function of $$p_i$$.  We only want infantesimal changes in probability to cause infantesimal changes in information.

4) If an event A_i is the intersection of two independent events $$B_i$$ and $$C_i$$, then the amount of information we gain when we find out C_i has happened should be a function of the intersection of $$A_i$$ and $$B_i$$. Also, the function should be equal to the information we gain when we find out $$A_i$$ and $$B_i$$ have happened.

Said another way, if $$P_1 = P_2 * P_3$$ then  $$I(P_1) = I(P_2) + I(P_3)$$.

Luckily, there is only one class of functions that fulfill these criteria:

$$I(event x) = k \log(p(x))$$

Now, k can be any negative number, so we pick k to give us units of bits.

This gives us:
$$I(event x) = \frac 1 {\ln(2)} * \ln(\frac1 {p(x)}) = -log_2(p(x))$$ where $$p(x)$$ is the probability of that event happening.

Now let's define a sort of measure of our surprise.  This is the information we gain from seeing the results of our predictions.  If the event happened, the knowledge we gained from our probability forecast was $$-log_2(f_t)$$.  However, if we picked incorrectly, we gained evidence for the prediction of the alternate event.  So, if we believed incorrectly, we gain $$-log_2(1 - f_t)$$ knowledge.

Let's work this out for a series of events:

We think there's a 10% chance of Bill winning an election in 2009.  Bill loses so we gain $$-log_2(0.9)\approx0.15$$ bits of info.  We gained very little information, because the 10% chance that Bill wins means we are close to certain that he will lose, and we were right.

We think there's a 90% chance of Bill winning his election in 2010.  Bill wins, so we gain  $$-log_2(0.9)\approx0.15$$.  Again, we gain very little information, because 90% is close to certainty.

Bill gets caught cheating on his wife with a goat before the 2011 election.  We now think there's a 1% chance of Bill winning.  He manages to win.  We are very surprised! We gain a lot of information this time.  We gain $$-log_2(0.01)\approx6.54$$.

Bill later turns out to have done a great job in office, despite the scandal.  We think there is a 90% chance that he gets reelected in 2012.  But we are surprised; he loses.  We gain $$-log_2(0.1)\approx3.32$$ bits of information.

Our total information gained for the 4 events is:
10.27 bits of information.

Our expected ignorance as a forecaster is about:
 $$\frac {10.27} { 4 }\approx2.57$$ bits per forecast.

Saturday, October 8, 2011

Retracto

Rising trend of retractions of scientific papers.

Monday, September 5, 2011

Market Efficiency and Government

If markets are weak form efficient then governments are unnecessary.
If markets (and thus people) are weak form inefficient then governments are a dangerous and bad idea.

Thursday, July 7, 2011

A directed graph of patent citations of the top 40 cited organizations (by unique patent assignee number) from 1980 to 2006

A directed graph of patent citations of the top 40 cited organizations (by unique patent assignee number) from 1980 to 2006.




Thursday, June 9, 2011

Why Pimps Wear so Much Jewelry. A lesson in asset protection.

Rick Harrison from the TV show "Pawn Stars" explains why pimps wear so much "bling". He says that when they get caught by the police the police confiscate all their money (which makes it impossible to pay bail), however they don't confiscate jewelry. The pimp can then pawn that jewelry again for 50% of what they bought it from the pawn shop in the first place and use that money for bail.

Thursday, February 17, 2011

Bitcoins Now Worth More Than 1 dollar.

Over the last month, the price of bitcoins has increased from roughly 40 cents to where now a bitcoin is now worth more than a dollar on the Mt. Gox exchange.

Saturday, February 12, 2011

An axe isn't enough.

Bruce Bartlett says the government is using an axe to cut the budget instead of a scalpel. Well, apparently with a 1.5 trillion dollar deficit out of a 4 trillion dollar budget, we need a chainsaw... an axe is simply too small.

Greg Mankiw Misses the Point

Greg Mankiw misses the point, it isn't a non-independent central bank that Ron Paul wants. It is independent money issuing banks that are not central.

Friday, July 16, 2010

BTW Sandpile

Here is my little sandpile model. I am trying to teach myself a bit about self organized criticality, so I coded this up.



Well, here is the code!

Note: I edited this post because I found a small bug in the original code.

#! /Library/Frameworks/Python.framework/Versions/Current/bin/python


# I ripped off the mencoder stuff from thee matplotlib website
# That stuff is copyright Josh Lifton 2004, used with permission
# The rest is copyright J. Emrys Landivar 2010
#
# 'Permission is hereby granted to use and abuse this document
# so long as proper attribution is given.'


import numpy as np
import matplotlib.cm as cm
import matplotlib.mlab as mlab
import matplotlib.pyplot as plt
import matplotlib.colorbar as cbar
from numpy.random import randn
from scipy import integrate
import matplotlib
matplotlib.use('Agg')
import subprocess                 # For issuing commands to the OS.
import os
import sys                        # For determining the Python version.



table_size=31 #the size of the table that the sandpile is on
z_crit=4  #the critical size of a stack of grains, it will fall if there are more grains that that
num_grains=1000 #how many grains to drop
z=np.zeros((table_size+2,table_size+2)) #thisis the sandpile and the table it rests on
add_location=(table_size/2+1,table_size/2+1) # sets the middle of the table to drop grains on

#
print 'Executing on', os.uname()
print 'Python version', sys.version
print 'matplotlib version', matplotlib.__version__

not_found_msg = """
The mencoder command was not found;
mencoder is used by this script to make an avi file from a set of pngs.
"""  #this was shamelessly ripped off from elsewhere

try:
    subprocess.check_call(['mencoder'])
except subprocess.CalledProcessError:
    print "mencoder command was found"
    pass # mencoder is found, but returns non-zero exit as expected
    # This is a quick and dirty check; it leaves some spurious output
    # for the user to puzzle over.
except OSError:
    print not_found_msg
    sys.exit("quitting\n")



z2=z.copy()
fig=plt.figure()


for g in xrange(num_grains):
    z[add_location[0],add_location[1]] += 1 #drop a grain
    for x in xrange(1,table_size+1): 
        for y in xrange(1,table_size+1):
            if z2[x,y] > z_crit:  #check for colapse
                    z[x,y]-=4   #colapse
                    z[x+1,y]+=1
                    z[x-1,y]+=1
                    z[x,y+1]+=1
                    z[x,y-1]+=1
    z2=z.copy()
    
    #drop the grains over the edge off the table.
    z[0,0:table_size+2]=np.zeros(table_size+2)
    z[0:table_size+2,table_size+1]=np.zeros(table_size+2)
    z[0:table_size+2,0]=np.zeros(table_size+2)
    z[table_size+1,0:table_size+2]=np.zeros(table_size+2)
    
    #plotting!!
    ax=fig.add_subplot(111)
    ax.set_title('Height of the Sandpile')
    cax = ax.imshow(z, interpolation='nearest')
    cax.set_clim(vmin=0, vmax=8)
    cbar = fig.colorbar(cax, ticks=[0,3, 5, 8], orientation='vertical')

    filename = str('%03d' % g) + '.png'
    plt.savefig(filename, dpi=100)
    print 'Wrote file', filename
    
    plt.clf()

#print sum(sum(z))

#make the movie

command = ('mencoder',
            'mf://*.png',
            '-mf',
            'type=png:w=800:h=600:fps=25',
            '-ovc',
            'lavc',
            '-lavcopts',
            'vcodec=mpeg4',
            '-oac',
            'copy',
            '-o',
            'output.avi')

#os.spawnvp(os.P_WAIT, 'mencoder', command)

print "\n\nabout to execute:\n%s\n\n" % ' '.join(command)
subprocess.check_call(command)

print "\n\n The movie was written to 'output.avi'"

print "\n\n You may want to delete *.png now.\n\n"

Saturday, July 3, 2010

Recalculation in the Manufacturing Labor Market

A new NY times story about how manufacturing jobs are having difficulty hiring people because of a skills shortage.  


Some random thoughts:
  • This seems to me to be a failure of education.
  • Eventually companies in the US will have to increase and fund their own education efforts (beyond just specific skills training), because it seems that our k-12 educational establishment has failed to teach a significant portion of our labor force.
  • This all fits Arnold Kling's "recalculation" story.


Sunday, May 16, 2010

Economic Alarm.

This is what people who are economically alarmed believe.



Do you guys think they are correct, or do you see hope?

h/t:  Daily Paul

Tuesday, April 27, 2010

Jerry Pournelle on on the Financial reform Bill

"Adam Smith told us that capitalist would always collude to get government to make it tough for competitors to enter the market place; the current financial reform bill does this in spades with big casino. Of course Sarbanes Oxley already did that; but this make the cheese more binding. If the goal is to lock in big combines and outfits too bid to fail, and be sure no one will compete with them and take some of their market share, the current "reform" bill is a very good continuation of the work started by Sarbanes Oxley."


I have to say, I couldn't agree more.

Wednesday, December 9, 2009

Back door free banking.

Could a few hedge funds decide to begin issuing debt denominated in the hedge fund's own debt?  If the funds became known for stability and their debt was highly rated, it would then function as a free bank.

Hrm, is an ETF functionally similar?

Update:  I don't know what I was thinking, an ETF is obviously not similar, because it pays creditors in currency not in its shares.

Monday, December 7, 2009

The two schools of economic thought.

There are really two big schools of modern economic though, the bottom-up and the top-down.  The bottom up school is made up of the Austrians and the New Institutional Economists.  The top-down school is made of the Monetarists and the new Keynesians.

Friday, December 4, 2009

A New Institutional Macroeconomics?

We had Keynsianism which was based on fiscal policy.  We have Monetarism which was based on monetary policy.  Will we have a new macro paradigm which is based on legal and institutional policy?

I was inspired by Scott Sumner's response to a comment about Japan's massive underground cash economy that I made on his blog.


Doc Merlin, I did my dissertation on currency hoarding and the underground economy.  The ratio of the tax rate to the interest rate is the number of years you can hoard income in cash before you would have been better off paying taxes. In the US that ratio correlates with the cash/GDP ratio over time. That ratio is high in Japan for two reasons; high taxes and low interest. So hoarding cash to evade taxes is profitable. Ironically, the liquidity trap may make their underground economy bigger.



... and also from De Soto's work.
If interest rates are low and taxes are high and if regulation is too onerous people on the whole will avoid the open economy and hide their activities. This will cause GDP will shrink. Transactions rates will slow in the formal economy, and go into the less efficient informal economy.  Formalness is not binary; it is a continuum.  For example if a company would like to go public to gain access to capital, but can't because of the expense of SarOx, it is being less efficient than it could be.  If chooses to avoid going public because the cost of complying with regulation is more than not complying, the entire future economy fails to realize the gains it would have had.

Also, now we have a good way out of a liquidity trap: tax cuts and economic deregulation.  (If the multiplier of fiscal stimulus is less than one as some studies are saying, its also a good idea to cut spending along with the taxes.)

Anything this gives me some examples of things that would help the US get out of a liquidity trap if we were to find ourselves in one:

  • Lowering minimum wage

  • Removing regressive taxes (Social security tax and medicare in the US, for example.)

  • Suspending Sarbane-Oxley



I wonder when this new macro will come about and what it will look like.

More observations from De Soto.

Hernando De Soto observes referring to the informal sector, "Whether you are inside the bell jar or outside, you will be taxed.  What determines whether you remain outside is the relative cost of being legal."

This is extremely important to a functioning government.  This means for a tax/regulation to function for revenue collection, it must be lower than the costs of succeeding in tax/regulation evasion plus the risks of punishment.

$$Functioning Tax or Regulation < costs of evading tax or regulation + subjective cost of the risk of punishment$$

Here we have the private benefits of the individual becoming a public bad under bad legal structure.  They hide their labor and pay bribes (which fund anti-productive activities).  Also the fact that they hide their labor means that the results of their labor are less accessible.

Thursday, November 5, 2009

The Mystery of Capital

I just started reading the book, "The Mystery of Capital" by Hernando de Soto, and it occurred to me that one of the functions of private property was to mostly concentrate risk and reward to the persons who were responsible for the action.   It seems obvious now that I think about it.

Definition: Expectations

I have to further define 'Expectations.' An expectation is something that an economic actor acts upon based on perception of the future. For example, a doomsday cult believes that the there is a UFO in the tail of the comet Hale-Bop. This UFO will take their spirits away to a better place when they die if they die while Hale-Bop is near earth. They then rationally proceed to commit mass suicide. They weren't insane or irrational, just horribly wrong.

I think I will change the name of this blog to 'Expectations Matter.'

Wednesday, November 4, 2009

Definitions:

I need a new word for "model" when referring to the ideas in the economic actors' heads that cause them to act certain ways. "Model" causes to much confusion because it seems to refer to more formal and academic types of thinking. Maybe "expectations" is better?

Ok, here goes:
Economic actors have expectations, when they act on their expectations and they turn out to be correct, these expectations are rewarded. When they are wrong they are punished.

Ok, I like this better.

Re:Does macro need a paradigm shift

This is in response to Scott Sumner's post on his blog, which I recommend reading.


1.

a. First of all, I agree that we need a common, more formal definition of what it means for money to be "tight" or "loose." I think it will be impossible to define what we mean by 'tight/loose money' because of the basic differences in paradigm between the different economic views. Its not that we are all on separate pages; we are reading out of different books.

b. I disagree that your views are that far from the norm, you are fairly standard for a monetarist school economist. Its rather that the econ-blogosphere tends towards the heterodox.

2. This plot suggests that the money supply wasn't that different than normal. AMB was same as normal at the start of the recession M1 and M2 were also.


3. CPI saw a small bump, but PPI had a massive spike.

So, I guess, yes, wrt actual commodities, money was tight, but in an absolute sense, money was not tight (M2 didn't drop appreciably).
That makes me think the 'tight money' story isn't true, because 'tight money' would cause deflation not huge PPI inflation.

4. M2 grew at a much faster rate than M1 (as would be expected) and AMB grew much faster than either of those. To me this looks like a huge push to de-leverage and get out of certain types of assets and get into cash. Also this fits the story of the Fed buying up bank assets.

5. I don't buy the TIPS story, precisely because there was a global push to get out of securities and into cash to avoid bankruptcies.

6. There is no six.

7. I think the 'recalculation' story explains this recession better than a 'tight money' story. Businesses made bad bets, their income was too low for their assets and then they had to de-lever, and switch to more profitable/less expensive business plans. This also somewhat fits John Geanakoplos's theory.

8. It seems to me that the Fed's basic problem is that it is ignoring markets in an attempt to fix them. It can't properly price interest rates which results in either gluts or shortages of money/credit.