The STEM job market appears to be a Catch-22 situation where almost the only way to get a job is to have at least two years, usually 3-5 years, of paid professional work experience. In a number of contexts, employers complain that they cannot find qualified STEM workers and that hundreds of thousands, even millions of job listings such as those surveyed allegedly go unfilled. Yet, this is not surprising. For example, for employers to be able to hire 10,000 STEM workers with 3-5 years of paid professional work experience, other employers must have hired 10,000 STEM workers with no experience, 3-5 years before. Only 2-5 percent of STEM job listings are for entry-level or junior-level (less than 2 years of experience) workers. Probably only about 1-2 percent of STEM job listings are for true entry-level workers — no experience. Probably about ninety-five percent of STEM job listings are for STEM workers with 2-10 years of work experience. Thus, it is not surprising that employers are unable to find STEM workers with work experience.

Remarkably, this strong preference for only 3-5 years of work experience prevails across many STEM fields and sub-fields, including fields such as molecular biology that would be expected to have very long, many-year learning curves.

These findings largely mirror the findings of Professor Peter Cappelli of Wharton Business School in his book Why Good People Can’t Get Jobs: The Skills Gap and What Companies Can Do About It and in various editorials that he has written and interviews that he has given.

This article is a followup to a previous article What is really hot in STEM jobs? This article backs up some of the statements in the previous article with a larger survey and hard numbers.

The Catch-22 Statistics Job Market

Statistics is by far the largest category of mathematical jobs in private industry. In this article, statistics job posts and jobs includes business analytics, data scientist, predictive modeling, and many “big data” jobs which are currently hot as well as more traditional highly statistical jobs such as actuaries in the insurance industry.

A survey of statistics jobs on LinkedIn revealed very few entry-level or even junior-level (less than two years of experience) jobs, negligible training opportunities, and a strong peaking in the range of 3-5 years of experience. The range of “3-5 years” of experience is the most common range (2-3, 2-7, etc.) requirement. Five years/at least five years of experience is the most common experience requirement for these jobs. As we shall see, remarkably, exactly the same numerical pattern holds for a wide range of different STEM jobs, seemingly independent of field or the complexity and likely learning curve of the subject matter.

Experience Levels for Statistics Job Posts
LinkedIn Anywhere Search (Feb. 23, 2013)

statistics                          4229

Entry Level and Junior Level
 
statistics "entry-level"              53
statistics "new grad"                 10
statistics "new college graduate"     39
statistics "recent college graduate"   4
statistics intern                    112
statistics "summer intern"            13
statistics "new phd"                   1
statistics junior                    175 ("junior" often used in other contexts -- not a job requirement)


Experience Levels in Years

statistics "1+ years"                 31
statistics "2+ years"                419
statistics "3+ years"                492
statistics "4+ years"                282
statistics "5+ years"                762
statistics "6+ years"                123
statistics "7+ years"                164
statistics "8+ years"                153
statistics "9+ years"                 11
statistics "10+ years"               216
statistics "11+ years"                 3
statistics "12+ years"                35
statistics "13+ years"                 2
statistics "14+ years"                14
statistics "15+ years"                69
statistics "16+ years"                 1
statistics "17+ years"                 2
statistics "18+ years"                 9
statistics "19+ years"                 0/4 (off LinkedIn site)
statistics "20+ years"                23
statistics "25+ years"                24
statistics "30+ years"                48

Ranges of Experience in Years

statistics  "2-5 years"               25
statistics  "2-7 years"                2
statistics  "3-5 years"              153
statistics  "3-7 years"                4
statistics  "5-7 years"               72


The Training Gap
statistics "will train"                3
statistics "training provided"         4

A Close Look at the SAS Job Market

I also took a look at the job market for a specific statistics-related skill, the SAS statistical analysis package, one of the mostly widely used statistical packages in private industry. The same patterns are found in SAS job posts. Three to five years of work experience is the most commonly requested range of experience. Five or at least five years of experience is the most common experience requirement. There are very few entry-level positions or evidence of employer provided training. Finally, job posts for ten or more years of experience are rare.

Saturday, Feb. 23, 2013 (8:21 pm PST)
LinkedIn "Anywhere" search


Key Word/Phrase           Number                 Notes (if any)


SAS                       2,460

SAS marketing               797
SAS finance                 510

SAS insurance               351
SAS actuarial                80
SAS actuary                  11

SAS real-estate              31
SAS economics               415
SAS econometrics             82

SAS health                  395
SAS medicine                 36
SAS medical                 363
SAS pharmaceutical          218
SAS clinical                296

SAS biostatistics           114 

SAS regulatory              310
SAS regulation               10


Overlap with Other Statistics Key Words and Phrases

SAS MATLAB                  145
SAS SPSS                    419
SAS statistics              858
SAS statistical           1,004
SAS "statistical analysis"  358
SAS "machine learning"       71
SAS "data mining"           332
SAS "text mining"             0
SAS analytics             1,040
SAS predictive              301
SAS "predictive model"       10
SAS "predictive models"      89
SAS "predictive analytics"   64


Credentials

SAS PHD                     178
SAS master's                390
SAS bachelor's              684
SAS BS                      261



Work Experience

SAS entry-level              33
SAS new grad                  4
SAS intern                   25

SAS "1+ years"               24
SAS "2+ years"              255
SAS "3+ years"              317
SAS "4+ years"              170
SAS "5+ years"              449

SAS "6+ years"               83
SAS "7+ years"              113
SAS "8+ years"              134
SAS "9+ years"               12
SAS "10+ years"             123

SAS "15+ years"              21
SAS "20+ years"              11

SAS "2-5 years"              25
SAS "3-5 years"              89
SAS "5-7 years"              34
SAS "3-7 years"               4
SAS "7-10 years"             10


The Training Gap

SAS "training provided"       0
SAS "will train"              2   (3-6 month training period)


Specific Statistical Methods

SAS Bayesian                 23
SAS Markov                    5
SAS likelihood                5
SAS coefficient               1    (GINI coefficient, quant-like job, lots of technical terms)
SAS kurtosis                  1    (quantitative researcher at Facebook) 
SAS Kolmogorov                0
SAS "neural networks"        26
SAS Pearson                   1    (company with name Pearson -- not a true match)
SAS Fisher                    3    (company names again -- not true matches)
SAS "robust statistics"       0
SAS "regression analysis"     0
SAS "least squares"           2
SAS stochastic               20

SAS support vector machine    3
SAS support vector machines   5
SAS "decision tree"          12
SAS "decision trees"         47
SAS Levenberg                 0    (as in Levenberg-Marquardt optimization)

SAS optimization            270    (often non-technical, non-statistics use)



High School and College Math (other than statistics)

SAS Calculus                  1    (quant job)
SAS Algebra                   2
SAS Trigonometry              0
SAS Geometry                  1

The Catch-22 Software Engineer Job Market

Remarkably, almost the same numerical and qualitative patterns can be found in job postings for general software engineering.

Experience Levels for Software Engineer Job Posts
Craig's List, All SF Bay Area, March 2, 2013  12:10pm

software engineer                     953
"work experience" software engineer   117   (experience explicitly qualified as "work experience")


Entry Level and Junior Level

"entry level" software engineer   15
"entry-level" software engineer    2
"0-2 years" software engineer      1
"junior" software engineer        45  (many due to phrases such as "Mentoring junior software engineers" or "guidance for junior team members")
"junior software engineer"         3
"new college grad" software engineer   0
"new college graduate"  software engineer  0
"recent college grad" software engineer 0
"recent college graduate" software engineer 4  (2 tutors, test engineer, technical support engineer)
intern software engineer          14
intern "software enginer"          6
"summer intern" software engineer  0
"software engineer intern"         3
"no experience" software engineer  3  (2 used "no experience" in another context; not a job requirement.  1: THE POSITION IS UNPAID)


Experience Levels in Years

"0+ years" software engineer       1 (Sr. Java Developer: Basic Qualifications:
• 5 Years with BS; 3 Years with Masters; 0 Years with PhD )

"1+ years" software enginer       17
"2+ years" software engineer     116
"2 years" software engineer      116 (appears to be same list as "2+ years")
"3+ years" software engineer     128
"3 years" software engineer      128 (appears to be same list as "3 years")

"4 years" software engineer       78
"5 years" software engineer      232
"6 years" software engineer       25
"7 years" software engineer       54
"8 years" software engineer       58
"9 years" software engineer        3
"10 years" software engineer      55  (significant number have language such as "our company has 10 years experience in xxx")
"15 years" software engineer      17  (significant number have language such as "our company has 15 years experience in xxx")
"20 years" software engineer       7  (significant number have language such as "our company has 20 years experience in xxx")


Ranges of Experience in Years

"0-2 years" software engineer      1
"3-5 years" software engineer     32
"3-7 years" software engineer      4
"5-7 years" software engineer     12
"10-20 years" software engineer    1 (Linux System Administrator job)
"10-15 years" software engineer    0


Experience Related Job Titles

"junior software engineer"             3
"senior software engineer"           124  (astonishingly "senior" often refers to jobs requiring 3-5 years work experience, even 7 years is rare)
"principal software engineer"         15  (often referes to higher levels of experience, such as 10 years, but not always)


The Training Gap

"training provided" "software engineer"   1    (PHP Programmer "Drupal experience or strong willingness to learn Drupal (training provided)")
"will train"  "software engineer"         0
"training provided" software engineer     1    (same PHP Programmer job)
"will train" software engineer            0

The Catch-22 Job Market for Scientists

Even more remarkably, the same patterns can be found for jobs labeled as “scientists.” These include many jobs in molecular biology which has proven extraordinarily complex, with cells containing complex networks of tens of thousands of genes and proteins. Nonetheless, employers seem convinced that only 3-5 years of experience in this extremely complex subject is needed to cure or effectively treat cancer or many other serious medical problems that have defied decades of heavily funded efforts.

Craig’s List

Scientist                                      74    (small majority are biotechnology and/or medical)

Scientist Job Posts by Craig's List Category

scientist ("Engineering Category")              4  (3 geologists, 1 alternative energy)
scientist ("food/beverage/hospitality jobs")    1  (product devlopment chef)
scientist ("education")                         3  (science teachers/tutors)
scientist ("business jobs")                     1  (Model Builder at XXX/Machine Learning/Big Data contract job)
scientist ("science jobs")                     39  (Mostly biotech/medical)
scientist ("healthcare jobs")                  11  (not quite same as "science/biotech" used on Web site)
scientist ("software jobs")                    11  (Big Data etc.)
(NOTE: Craig's List categorization of the jobs is somewhat confusing unfortunately.  "science jobs" and "science/biotech" appear to the same category.)


Scientist Types by Key Word or Key Phrase

scientist clinical                             24
scientist biology                              16
scientist clinical biology                      6  (check overlap)
scientist biotech                               7
scientist biotechnology                         2
scientist health                               19
scientist health clinical                      13  (check overlap)
scientist clinical biotech                      3


Entry Level and Junior Level

"entry level" scientist                1
"entry-level" scientist                0
intern scientist                       1
"new college grad" scientist           0
"new college graduate" scientist       0
"recent college grad" scientist        0
"recent college graduate" scientist    0
"junior scientist"                     0


Experience Level

"0+ years" scientist       0
"1+ years" scientist       2
"2+ years" scientist      10
"3+ years" scientist       4
"4+ years" scientist       6
"5+ years" scientist      15
"6+ years" scientist       1  (spurious match: "XXX has been in business for over 6 years")
"7+ years" scientist       5

"8+ years" scientist        0
"9+ years" scientist        0
"10+ years" scientist       4  (all science/biotech jobs, 8-10 years in one, 10 years preferred (5+ required), at least 10 years in two -- one of which is a managent position)
"11+ years" scientist       0
"12+ years" scientist       0
"13+ years" scientist       0
"14+ years" scientist       0
"15+ years" scientist       0

"20+ years" scientist       1  (spurious match: "...Founded more than 20 years ago")


Ranges of Experience Level

"0-2 years" scientist         0
"2-3 years" scientist         0
"3-5 years" scientist         3
"2-7 years" scientist         0
"3-7 years" scientist         0
"4-7 years" scientist         0
"5-7 years" scientist         4

The Training Gap

"will train" scientist         0
"training provided" scientist  0


NOTE: On Craig's List SF Bay Area plus sign appears to be ignored so that searches for "3+ years" and "3 years" return the same matches.

Scientist Job Posts on the LinkedIn Job Board

A very similar pattern was found with the “Scientist” job postings on the LinkedIn Job Board.

Scientist Jobs Posts
LinkedIn March 2, 2013, 6:28pm

scientist      1311

Health/Medicine/Biotech

scientist biology          224
scientist clinical         383
scientist medical          340
scientist biotech           57
scientist biotechnology    179
scientist health           332
scientist pharmacology      58
scientist genomics          21
scientist bioinformatics    26
scientist genetics          36


Analytics/Data Scientist/Big Data

scientist "machine learning"      163
scientist analytics               159
data scientist                    722
scientist "big data"               85
scientist "information retrieval"  36


Energy/Alternative Energy

scientist battery                 13
scientist "solar power"           2
scientist energy                  158  (a lot of spurious matches for some reason)
scientist "alternative energy"    0 / 150 (off LinkedIn site)
scientist nuclear                 13
scientist tesla                   0 / 56 (off LinkedIn site, mostly Tesla Motors jobs)
scientist bigelow                 0 /  3 (off LinkedIn site, none related to Robert Bigelow or Bigelow Aerospace)


Check Some Overlaps

scientist health clinical          166  (check for overlap)
scientist health medical           144
scientist clinical medical         228


Entry and Junior Level

"entry-level" scientist             10
"entry level" scientist             10  (same 10 as "entry-level" as expected for LinkedIn)

"new college grad" scientist         1   (data scientist)
"new grad" scientist                 4 
"new college graduate" scientist     0 / 472 (off site)
"recent college grad" scientist      0 / 276 (off site)
"recent college graduate" scientist  0 / 276 (off site)
scientist intern                     9
scientist "summer intern"            0 / 1000 (off site)
scientist "no experience"            1   (NOTE: "PhD with no experience.  ")


Experience Levels

"0+ years" scientist                   0 / 1000 (off site)
"1+ years" scientist                  16
"2+ years" scientist                 123
"3+ years" scientist                 132
"4+ years" scientist                  56
"5+ years" scientist                 211
"6+ years" scientist                  50
"7+ years" scientist                  49
"8+ years" scientist                  62
"9+ years" scientist                   6
"10+ years" scientist                 81
"11+ years" scientist                  2
"12+ years" scientist                 11
"13+ years" scientist                  3
"14+ years" scientist                  5
"15+ years" scientist                 32
"16+ years" scientist                  2
"17+ years" scientist                  0 / 110 (off site)
"18+ years" scientist                  6
"19+ years" scientist                  0 / 95 (off site)
"20+ years" scientist                 23
"25+ years" scientist                 27  (many uses such as "In 25 years, XXX has become a leading biopharmaceutical company...")
"30+ years" scientist                 11  (ditto)


Job Titles Related to Experience

"associate scientist"                 46
"senior scientist"                    86
"principal scientist"                 70
"lead scientist"                      13


Ranges of Experience

"0-2 years" scientist                 10
"2-3 years" scientist                  6
"3-5 years" scientist                 39
"5-7 years" scientist                 15
"3-7 years" scientist                  3


The Training Gap

"training provided" scientist          0 / 345  (off site)
"will train"        scientist         11

More on the Training Gap

I took a closer look at the use of the terms “apprentice” and “apprenticeship” which traditionally represent training programs for gaining work-based experience.

Apprentice and Apprenticeship
LinkedIn Anywhere Search
Sunday, March 3, 2013 10:21 AM (PST)

apprentice                           116
apprenticeship                       245

statistics apprentice                  1
statistics apprenticeship              5
statistics                          4209

scientist apprentice                   0 / 100 (off site)
scientist apprenticeship               0 /  76 (off site)
scientist                           1303

software engineer apprentice           6
software engineer apprenticeship      10
software engineer                 17,977

(NOTE: SQL refers to Structured Query Language, the most common database programming language)
SQL apprentice                         4
SQL apprenticeship                     3
SQL                               16,353

Conclusion

The key findings of this survey are that there are very few job posts for entry-level or even junior-level STEM positions (less than two years of paid professional work experience). There is very little evidence of training programs, apprenticeship programs, or other means by which entry-level workers such as new college graduates could acquire work experience. The most common range of work experience sought by STEM employers is 3-5 years independent of field, even in fields such as molecular biology that have proven exceptionally complex. The most common single number work experience requirement is five (5) years of experience, at least five (5) years of experience, or “5+” years of experience. It is also quite rare to find job posts seeking more than ten (10) years of work experience.

The STEM job market appears to be a Catch-22 situation where almost the only way to get a job is to have at least two years, usually 3-5 years, of paid professional work experience. In a number of contexts, employers complain that they cannot find qualified STEM workers and that hundreds of thousands, even millions of job listings such as those surveyed allegedly go unfilled. Yet, this is not surprising. For example, for employers to be able to hire 10,000 STEM workers with 3-5 years of paid professional work experience, other employers must have hired 10,000 STEM workers with no experience, 3-5 years before. Only 2-5 percent of STEM job listings are for entry-level or junior-level (less than 2 years of experience) workers. Probably only about 1-2 percent of STEM job listings are for true entry-level workers — no experience. Probably about ninety-five percent of STEM job listings are for STEM workers with 2-10 years of work experience. Thus, it is not surprising that employers are unable to find STEM workers with work experience.

One of the most remarkable aspects of this strong preference for STEM workers with only 3-5 years of work experience is that it appears to exist across a wide range of different STEM fields, including fields such as molecular biology that are exceptionally complex and almost certainly require decades to master at a purely technical level. Similar comments could be made about some highly mathematical fields of computer software such as video compression or speech recognition. Nonetheless, three to five years rules.

© 2013 John F. McGowan

About the Author

John F. McGowan, Ph.D. solves problems using mathematics and mathematical software, including developing video compression and speech recognition technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Resources and Suggested Reading

If there is a skills gap, blame it on the employer.
by Peter Cappelli
New York Times
August 3, 2012

Why Good People Can’t Get Jobs
By Peter Cappelli

Peter Cappelli Video Interview on PBS A Need to Know

Peter Cappelli Audio Interview on IEEE Spectrum Podcast

STEM jobs and education is the subject of a great deal of press coverage and public comment. There are perennial claims of shortages of STEM workers and that the United States is behind or falling behind other nations in STEM education. See for example the 2005 report Rising Above The Gathering Storm: Energizing and Employing America for a Brighter Economic Future and its 2010 followup report Rising Above the Gathering Storm, Revisited: Rapidly Approaching Category 5

This article surveys STEM job postings on the Craig’s List San Francisco Bay Area job board on January 13 and January 19, 2013. The key results of this brief survey are: very few entry-level or junior STEM jobs are posted. Most STEM job posts claim to require 2-7 years, often 3-5 years, of work experience in specific skills such as programming languages such as MATLAB or C++ and toolkits such as OpenCV or OpenGL. Very few jobs requesting more than 10 years of experience are posted. There is little or no interest indicated in a range of mathematical skills taught in high school and college math including algebra and especially calculus.

STEM job posts with a high mathematical content are dominated by statistics and data analysis, primarily business data and some medical/healthcare data. Most machine learning, “big data”, and data scientist posts fall into the category of statistics and data analysis. There are remarkably few STEM jobs posts seeking to solve “big problems” such as alternative, cheaper energy sources, curing major diseases such as cancer, and the like. The few companies that arguably post “big problem” jobs are often backed by the government (e.g. Elon Musk’s Tesla Motors and SpaceX) or may have special relationships with the government (e.g. Robert Bigelow’s Bigelow Aerospace).

The General STEM Market

In many respects, STEM jobs in the San Francisco Bay Area overlap heavily with software and programming jobs. To establish the context, the author took a look at software jobs with various requirements.

Craig's List San Francisco Bay Area Jan 13, 2013


Search Phrase           Number Found

"programmer"             107
"software engineer"          371
C++ "software engineer"        112
java "software engineer"       213
python "software engineer"      108
C# "software engineer"        45
embedded "software engineer"     27
phd "software engineer"        22
"junior software engineer"      1
"junior programmer"          1
"entry level software engineer"    0
entry level "software engineer"    3
junior "software engineer"      16
"new college graduate" "software engineer"  0
"senior software engineer"      98
C++ "senior software engineer"    36
java "senior software engineer"    68
python "senior software engineer"   35 
C# "senior software engineer"     15
embedded "senior software engineer"  4

"software architect"        11
"principal software engineer"   6
"software project manaager"    1

Training
training "software engineer"       29
"will train" "software engineer"     0
"training provided" "software engineer"  0 

"data scientist"          6
"algorithm engineer"        3
"algorithm scientist"       1

PHD                127
PHD MATLAB             8
PHD OPENCV             0

Programming Languages
C++                234
C#                 131
java                624
python               385
PHP                406
Ruby                256
Perl                246
SQL                537
MySQL               335
"Objective C"            81
"Objective-C"            46
MATLAB               35
Mathematica             2
SAS                 30

Functional Languages/Exotic Languages
haskell               0
erlang                2
lisp                 1 (job at Apple)
lua                 19
ocaml                0

Version Control

Git                 113
Subversion             47
CVS                 32
Perforce              24
Mercurial              4
RCS                 2 (doubtful computer)
SourceSafe              1


Toolkits etc.
OpenCV  (Computer Vision)             4
OpenGL  (3D Computer Graphics)          17

Mobile
iphone               197
Android              260
xcode               17

Miscellaneous
database              1133
emacs               1
Linux               525
hadoop               105

There are two things that stand out. First, even in the Bay Area and even with the current hype about mobile devices, software jobs are dominated by tools used for business applications, especially the processing and analysis of business data: databases as a general category, SQL (Structured Query Language used by many databases), etc. Secondly, junior or entry level jobs requiring no or even less than two years of experience are extremely rare!

A Catch-22 STEM Job Market

One may ask if companies hire very few entry-level STEM workers, where do experienced STEM workers come from? Indeed, in some contexts, companies frequently claim to have large numbers of job openings in STEM fields that are never filled. But this is hardly surprising given that these job openings are for experienced STEM workers only and the companies, at least based on job posts found on Craig’s List and other job boards (which are similar), rarely hire entry-level STEM workers.

A Shortage of Calculus Tutors?

A great deal of mathematics education in high school and college in the United States is in algebra and calculus. When corporate titans and big name academics on blue ribbon panels bemoan the state of STEM education in the United States they seem to be saying that the United States is not producing enough students with the critical algebra and calculus skills that employers are struggling to find. Algebra and calculus being the mainstays of high school and college math education.

Algebra and Calculus in the Real World

algebra         79 (50 in education category)

algebra tutor      40
algebra teacher     12
algebra instructor    7

"linear algebra"     6 (2 tutors, 1 machine learning, 3 repeats of "Senior 3D Physics Engineer" -- game company/perpetual advertiser)


"calculus"             61  (at least 55 tutors/instructors/teachers/etc.)
-- 59 in education category on sat jan 19, 2013) out of 63 in "all jobs"
other 4 were cross-posts of tutor/writer/etc. posts also in education/teaching category
 

Remarkably, all calculus jobs found on Saturday, January 19, 2013 were calculus tutors, teachers, and other forms of educators. A large majority of algebra jobs were tutors, teachers, and educators. Linear algebra which is usually taught in first or second year college mathematics courses and which arguably has considerable practical use turned up in a only a few job posts. STEM topics such as linear algebra tend to turn up most frequently in computer game job posts, not exactly what most people think of when they hear the United States is falling behind (Russia/Japan/China/India/fill in your favorite foreign bogey man here) in STEM education and workers!

The Wonderful World of Statistics

However, if we look at statistics, machine learning, data science, and similar topics, we do find many job posts, primarily for statistical data analysis of business data and sometimes medical/health data.

January 19, 2013 search results

statistics       263 (37 in education/teaching category)
statistics software   83 (2 in education/teaching category)
"data scientist"     6
statistics "data scientist" 4 (check overlap)
"machine learning"      59
statistics "machine learning" 12 (check overlap) 
statistician       1
SAS           40
SPSS           31
"statistical package"  1
"such as R"       5 (all refer to R programming language)

"big data"        155 (0 in education/teaching category)
"big data" hadoop     64
"big data" hbase     31
"big data" statistics   13
weka            2

NOSQL           95
nosql "big data"     21

probability       15
 

Scientist Mostly Means Biotech

A broader survey of STEM-related keywords and phrases showed that “scientist” in a job title mostly refers to biotechnology, health care, and medicine jobs. Every now and then one may encounter a “data scientist,” an “algorithm scientist,” or something like that, but most scientist positions are biology and medicine jobs — at least on Craig’s List San Francisco.

STEM KEYWORDS AND PHRASES

"calculus"             61  (at least 55 tutors/instructors/teachers/etc.)
-- 59 in education category on sat jan 19, 2013) out of 63 in "all jobs"
other 4 were cross-posts of tutor/writer/etc. posts also in education/teaching category

(saturday jan 19 2013 search)
scientist 54 (vast majority science/biotech or healthcare)
handful of exceptions:
 -- Sr. Data Scientist
 -- Speech Scientist
"data scientist" 6
"speech scientist" 2
Scientist/Research Engineer, Applied Science 1 (basicaly data scientist)
scientist teacher  1

engineer       1318 (mostly software)
"research engineer"   3
"electrical engineer" 35
"mechanical engineer" 36
"chemical engineer"   2

physicist        1
physics        131 (62 in education category)
physics tutor      42
physics teacher     13
physics instructor   12
physics faculty     7 (lot of overlap with tutor/teacher/etc.)
physics game      15 (computer game jobs on examination)
physics engineer    41 (gaming/optical engineering/misc/not tutor)

mathematics       138 (44 in the education category)
"applied mathematics"  5  (3 in education/teaching category)
mathematician      8 (6 repeats of same web job, teacher, algorithm scientist)

mathematics teacher   5
mathematics tutor    17
mathematics instructor 9
mathematics professor  1 (visiting assistant professor of statistics)
mathematics faculty   15
mathematics educator  3

mathematics software  59 (vast majority are software engineer jobs)
mathematics phd     8

algebra         79 (50 in education category)

algebra tutor      40
algebra teacher     12
algebra instructor    7

"linear algebra"     6 (2 tutors, 1 machine learning, 3 repeats of "Senior 3D Physics Engineer" -- game company/perpetual advertiser)

statistics       263 (37 in education/teaching category)
statistics software   83 (2 in education/teaching category)
"data scientist"     6
statistics "data scientist" 4 (check overlap)
"machine learning"      59
statistics "machine learning" 12 (check overlap) 
statistician       1
SAS           40
SPSS           31
"statistical package"  1
"such as R"       5 (all refer to R programming language)

"big data"        155 (0 in education/teaching category)
"big data" hadoop     64
"big data" hbase     31
"big data" statistics   13
weka            2

NOSQL           95
nosql "big data"     21

probability       15

"mathematical mode"   0

theorem         0

Big Problem STEM Job Posts Are Rare!

Very few job posts involve solving or attempting to solve “big problems” such as new, alternative, cheaper energy sources, power and propulsion systems, or major medical problems. One occasionally finds jobs related to solar power or batteries on Craig’s List San Francisco. Only a few of these are engineering or R&D jobs. These often are companies backed by the government such as the now infamous Solyndra. It is very rare to see anything related to nuclear or thermonuclear fusion power.

Elon Musk’s Tesla Motors and SpaceX (Space Exploration) advertise jobs from time to time on Craig’s List. Both companies are heavily backed by the United States federal government. Probably the most exotic “big problem” job posts that appear on Craig’s List are from Las Vegas hotel billionaire Robert Bigelow’s Bigelow Aerospace, some of which read like X-Files episode plots. Bigelow Aerospace has received contracts from the federal government and there is considerable speculation regarding Robert Bigelow’s relationship with parts of the government.

BIG PROBLEMS  Saturday Jan 19, 2013

"solar power"   67 (most non technical)
engineer "solar power"  3
scientist "solar power"  0
solar 232
solar scientist 0
battery engineer 6 (not battery R&D though)
"battery engineer" 0

"fusion power"  0
"thermonuclear fusion" 0

"alternative energy" 10

tesla motors      3 (two clerical, 1 IT app -- not car engine developer etc.)
bigelow aerospace   2 (dated Dec. 28, 2012)
 -- chemical engineera
 -- mechanical engineer (spaceship life support systems?)

Conclusion

The key results of this brief survey of the Craig’s List San Francisco Bay Area job board are: very few entry-level or junior STEM jobs are posted. Most STEM job posts claim to require 2-7 years, often 3-5 years, of work experience in specific skills such as programming languages such as MATLAB or C++ and toolkits such as OpenCV or OpenGL. Very few jobs requesting more than 10 years of experience are posted. There is little or no interest indicated in a range of mathematical skills taught in high school and college math including algebra and especially calculus.

STEM job posts with a high mathematical content are dominated by statistics and data analysis, primarily business data and some medical/healthcare data. Most machine learning, “big data”, and data scientist posts fall into the category of statistics and data analysis. There are remarkably few STEM jobs posts seeking to solve “big problems” such as alternative, cheaper energy sources, curing major diseases such as cancer, and the like. The few companies that arguably post “big problem” jobs are often backed by the government (e.g. Elon Musk’s Tesla Motors and SpaceX) or may have special relationships with the government (e.g. Robert Bigelow’s Bigelow Aerospace).

Many of these results contradict common claims and themes in general news media articles, think tank reports, and other sources about STEM education, STEM employment, and alleged shortages of STEM workers (scientists, technologists, engineers, and mathematicians).

It is common in the business press, as well as conservative and libertarian sources, to see claims that the government and/or politicians are short-sighted compared to the long term strategic vision of private industry. This is remarkably unsupported by the hiring patterns found on Craig’s List (and many other job boards, which are similar in content). Private businesses seem remarkably uninterested in tackling serious problems such as energy despite soaring prices and evident problems. In the rare cases where someone may be attempting to solve these problems, one often finds the heavy hand of government funding, for better or worse. Whatever one thinks of the much ballyhooed fracking technology, this, on close examination, can be traced to US Department of Energy research funding, and not, for example, the photogenic ExxonMobil scientists featured in ExxonMobil’s 2008 ad campaign.

© 2013 John F. McGowan

About the Author

John F. McGowan, Ph.D. solves problems using mathematics and mathematical software, including developing video compression and speech recognition technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Doctor Who (played by Tom Baker), from The Face of Evil (original airdates: 1 January – 22 January 1977)

Introduction

L. Gordon Crovitz of the Wall Street Journal recently posted an op-ed piece Gordon Crovitz: Who Really Invented the Internet? Contrary to legend, it wasn’t the federal government, and the Internet had nothing to do with maintaining communications during a war. that generated widespread controversy and several detailed rebuttals: No credit for Uncle Sam in creating Net? Vint Cerf disagrees. A legendary figure in the invention of the Internet weighs into a new debate about the U.S. government’s role during that heady era., So, who really did invent the Internet? by Michael Hiltzik, and many others. Of course, Crovitz’s claims are factually inaccurate — extremely inaccurate.

In fact, the US Government — and European governments through their support of the European CERN laboratory — played a major and dominant role in the invention of the Internet. Since World War II, governments have become the major and in many cases sole source of funding for most basic and much applied research in the world, not just the United States. This represents a radical change over the past, prior to World War II, when private individuals and organizations performed a substantial fraction of basic and applied research. Government funded research has had some spectacular practical successes: the Manhattan Project that developed the first nuclear power plants and atomic weapons, the development of the hydrogen bomb, aviation and rocketry through the early 1970s, and, of course, the Internet. There have been other lesser successes as well.

However, as many including Peter Thiel, Tyler Cowen, and (yes!) Paul Krugman have noted, since about 1970, we have made remarkably poor progress in many areas outside of computers and electronics, notably in power and propulsion. This is one of the main reasons that the term “technology” has become increasingly synonymous with “computer and electronic technology” in popular English usage. Where are the flying cars?Why has the expenditure of about $200 billion inflation adjusted dollars, about ten times the inflation adjusted budget of the Manhattan Project, failed to produce any significant advances in the treatment, let alone cure of most cancers? Many other examples can be cited.

Despite the “free market,” anti-government rhetoric of much of the business community — epitomized by the Wall Street Journal editorial page and taken to ludicrous extremes in Crovitz’s article, modern “private” companies often depend heavily on the government to research and develop the core science and technology for their products and services. This is very true in the computer and Internet industry, which ironically is held out falsely as an ideal example of the “free market” at work. What in fact occurs is that if the government research program is on the right track, then private industry has usually been able to exploit the results of the research and progress occurs.

Where the government research programs are stalled, private industry has been unable and, in most cases, unwilling to find an alternative approach that works. This is not simply a matter of willpower. Private individuals and organizations have lost much of the expertise in basic research that was more common before World War II. Consequently when private individuals like former Intel CEO Andy Grove, frustrated with the poor results of government research on cancer and Parkinson’s disease, attempt to find alternatives to stagnant mainstream research programs, they often fail.

The basic research methods popular in the government research no longer include some of the methods used prior to World War II, with, for example, government research mega-projects like the ITER tokamak fusion reactor or the LHC displacing the inexpensive table-top methods of the past in many fields. It can, in some cases, even be difficult to determine what the inventors and discoverers of the past were doing. How significant was the frequent interest in and self-reported use of philosophy and even sometimes the occult found in pre-World War II inventors and discovers such as Johannes Kepler, Isaac Newton, Niels Bohr, Albert Einstein, Wolfgang Pauli, rocket pioneer Jack Parsons (one of the most extreme examples), and others? Thus, it is difficult for private individuals or organizations to recreate some types of expertise that may be needed once again to solve pressing problems like energy shortages or cancer.

There Can Be Only One

For a number of social and political reasons government research programs frequently become fixated on a single or small number of narrowly defined theories and approaches, defend these theories and approaches stubbornly and effectively despite repeated lack of practical results, and successfully market these theories and approaches to policy makers, the general public, and business leaders — again despite the lack of practical results. This has led, I strongly suspect, to stalled or severely slowed progress in many fields compared to the pre-1970 and especially the pre-World War II period. I will present a theory as to why the slowdown occurred in the 1970′s rather than in World War II and has continued since then.

I use the term theory to refer to unifying fundamental concepts such as superstrings in theoretical physics, the oncogene theory of cancer in the War on Cancer, and the Big Bang theory in cosmology. I use the term approach to refer to specific technical approaches or methods such as the heavy reliance on Monte Carlo simulations in experimental particle physics, the use of immortal cell lines in cancer research, and tokamaks in fusion power research.

[ReviewAZON name="The Trouble with Physics" id="6" display="inlinepost" asin="061891868X" trackingid="antoniocangia-20" country="us" width="200px" float="left" imagetop="10px"] One of the better known and also one of the more widely criticized examples of this dominance by a single theory is the preeminent position of superstrings theory in theoretical physics since about 1984 (which actually replaced the non-Abelian Gauge Grand Unified Theory (GUTs) fad that dominated from 1974 until the mid 1980′s when the failure to find proton decay presented a serious problem for the GUTs). From what I understand, this dominance has declined somewhat following a barage of negative publicity such as Lee Smolin’s The Trouble with Physics, Peter Woit’s popular Not Even Wrong blog and book , Lawrence Krauss’s Hiding in the Mirror, various comments by Roger Penrose, and other criticisms such as Sheldon Glashow‘s well known antipathy to string theory.

[ReviewAZON name="Hiding in the Mirror" id="8" display="inlinepost" asin="0143038028" trackingid="antoniocangia-20" country="us" width="200px" float="left" imagetop="10px"]

Superstring theory is particularly striking because it had and has no basis in experimental data or evidence. It is simply an example of a “good idea” that somehow either squeezed out or subsumed its competitors. Supersymmetry, for example, was integrated into superstrings along with a number of other theoretical ideas that predated the superstrings synthesis. From a social and political point of view, one of the advantages of superstrings is that it provided a unified framework that could integrate the efforts of several previously competing groups of theoretical physicists in a single group effectively led by Ed Witten and a few other senior theoretical physicists. One big group is usually able to lobby the government, the US Congress, and the public for support more effectively than several medium-sized groups, let alone a large number of competing individuals and small groups.

Superstrings is not an isolated case. In the early 1970′s, during the first “energy crisis,” tokamaks squeezed out a large number of other competing approaches to developing nuclear fusion power. In this case, early promising results with tokamaks were heavily hyped and used to lobby both for greatly increased overall funding for fusion power research and to kill other competing approaches. Forty years later, we are still waiting for the advent of fusion power and the ITER fusion mega-project promises little in practical usable results.

So too, in the mid 1970′s the failing retroviral theory of cancer transformed into the oncogene theory of cancer and rapidly crushed all competing theories of the dread disease, quickly garnering a Nobel Prize and truly massive funding. The US National Cancer Institute alone has an annual budget of over $5 billion today, adjusting for inflation the same annual budget as the wartime Manhattan project. This brief account probably understates the extent to which certain approaches such as immortal cell lines came to dominate both the theory and methodology of cancer research.

In the early 1970′s, the US Defense Advanced Research Projects Agency (DARPA) conducted a contest between different approaches to speech recognition which was won by a team from Carnegie-Mellon University using a method that is now known as the Hidden Markov Model or HMM speech recognition. Essentially all speech recognition engines and research today is now descended from this one approach. Although performance has improved it still leaves much to be desired and progress has been slow at best.

Many, many other examples of this process exist. In most cases, in the 1970s or early 1980s, a single theory or approach that showed early promise on experimental or theoretical grounds or both was hailed as the answer with the desired results — a theory of everything (TOE) in physics, cheap energy in fusion power, a cure for cancer, talking computers — just around the corner. In some cases like the War on Cancer and fusion power, actual funding increased substantially. In most cases, whether funding increased or not, the single theory captured the lion’s share of funding, pushing out all or most competitors. Government research programs became much more narrowly focused in the 1970s and 1980s and have remained so since. In most cases, the promised results never came at all or have fallen far short of initial expectations.

The Ph.D. Glut and the Decline of Scientific Productivity

The mid-twentieth century and World War II in particular saw a transformation of science and research and development. Funding both greatly increased in quantity and shifted to almost exclusively the government. Science became much more professionalized and institutionalized. The role of so-called “amateurs” declined dramatically. The importance of the Ph.D. and other credentials increased markedly. The success of the wartime Manhattan Project spawned numerous attempts to duplicate the success of this giant project; most (not all) of these attempts have so far failed.

These dramatic mid-century changes led to a period of exponential increase in the funding for many kinds of scientific research, especially during the decade following Sputnik (October 4, 1957). This led to an exponential increase in the number of Ph.D’s in many, many fields. For a time, especially after Sputnik, most of these freshly minted Ph.D.s could find jobs in their profession. The bubble burst in the late 1960′s. This produced a huge glut of Ph.D.’s, would-be scientific researchers, since about 1970.

Government, business, leading scientists, and (yes) the Wall Street Journal editorial page have lobbied long and hard — up to the present day — to maintain and, if possible, expand this ocean of Ph.D.’s who cannot find jobs in their supposed professions (see the examples in Appendix II below). One official study after another claims there is a shortage or imminent shortage of scientists in the United States, strikingly at variance with actual reality where most Ph.D. recipients end up doing something other than science, with a lucky few leaving science to develop iPhone apps that sell dogfood and similar gimmicks — anything but actually curing cancer or finding new, cheaper energy sources or solving other pressing problems.

It is likely that the end of the exponential increases in government funding for research and development coupled with the perpetual Ph.D. glut since the late 1960′s account for the rise of the knowledge monopolies in which a single theory and/or approach acquired an extremely dominant position — usually in the early 1970s. With dozens of Ph.D.s competing for each available position, those who organized into the largest and most regimented groups often won out in the political battle for resources within the government. Bigger may not be better but it can lobby for funding much more effectively.

With far more Ph.D.’s than actual jobs, it is trivial to eliminate, consciously or not, anyone who dares to question the One Right Way. Conformity is highly rewarded and true risk-taking usually (not always) career suicide. Ironically, scientific and technological progress would likely have been higher if there had been a true shortage of scientists, as repeatedly falsely claimed over the last forty years. Then, the scientists would have enjoyed the creative freedom and independence so often necessary for major scientific and technological progress.

Conclusion

So we come full circle back to the Internet and computers. Yes, the government did invent the Internet. Yes, government funded research and development has had and may continue to have great successes. People like Gordon Crovitz who deny this are, at best, living in a free market fantasy world.

However, most government research programs have not been very successful despite vast funding. Most attempts to replicate the remarkable success of the Manhattan Project have failed. The rate of progress in many fields appears to have slowed since about 1970. Perhaps this is just the luck of the draw as Paul Krugman has suggested. It is more likely that the slowdown has been caused by the drawbacks of centralized funding and control of scientific research, the dominance of many fields by one or a few narrowly focused theories and sets of approaches, and the many negative effects of the Ph.D. glut.

What can we do to improve the rate of scientific progress and solve the problems such as energy shortages that probably contribute greatly to the current global economic slowdown and seeming conflicts over oil and energy resources? One may wonder about the true nature of seeming conflicts over energy resources that result in less energy and higher energy prices as almost certainly occurred with the wars in Iraq and Afghanistan, but that belongs in a different discussion.

The private sector can step up as Andrew Grove and Peter Thiel have tried to do to provide an alternative to the current government funded research programs that have so far failed to produce much, if any, progress in many fields. It is important to remember that the private sector has lost expertise in basic research. We do not live in the era of colorful private inventors like Octave Chanute and the Wright Brothers. Rather most Silicon Valley businesses have expertise in commercializing proven technologies originally developed in the government research programs. This is not the same as the basic and applied research that leads to a proven technology.

In principle, the federal government can move to break up the huge centralized research institutions and replace them with a more decentralized research funding system, such as many science foundations instead of one National Science Foundation. I would not hold my breath for this one.

The National Academies of Sciences report Bridges to Independence: Fostering the Independence of New Investigators in Biomedical Research contains a number of other suggestions that seem worthy of consideration.

Ironically, creating an actual scientist shortage by intentionally reducing the number of Ph.D’s produced annually at government expense below the number of senior scientists who die or retire each year might yield a significant increase in the rate of scientific progress.

© 2012 John F. McGowan

About the Author

John F. McGowan, Ph.D. solves problems using mathematics and mathematical software, including developing video compression and speech recognition technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Appendix I: Selected Internet Resources on the Ph.D. Glut

“U.S. pushes for more scientists, but the jobs aren’t there” by Brian Vastag in the July 7 Washington Post.

How and Why Government, Universities, and Industry Create Domestic Labor Shortages of Scientists and High-Tech Workers
By Eric Weinstein

Toil, Trouble, and the Cold War Bubble: Physics and the Academy since World War II
MIT Science Historian David Kaiser Presentation at the Perimeter Institute

PhD’s Org’s Collection on Links and Resources on Scientist and Ph.D. Shortage Claims

Bridges to Independence: Fostering the Independence of New Investigators in Biomedical Research (National Research Council 2005)

Women in Science by Philip Greenspun

Appendix II: Selected Internet Examples of the Never Ending STEM Shortage Claims

Our Ph.D. Deficit (Op-Ed by Norman Augustine and Burton Richter, Wall Street Journal, May 4, 2005)

Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future (2005)

Heading Off a Ph.D. Shortage. by John Vaughn and Robert Rosenzweig in Issues in Science and Technology, v7 n2 p66-73 Win 1990-91

EDUCATION; Shortage of Ph.D.’s Imminent, Report Says (January 24, 1990)

If you’ve always wanted to start your own technical blog, perhaps about a mathematical or scientific topic, but never got around to do it or failed to attract a following, read on.

My book about technical blogging is now available in Beta from The Pragmatic Bookshelf:
http://pragprog.com/book/actb/technical-blogging

It’s the kind of book that will teach you everything you need to realistically know to succeed at blogging. In it I provide a complete road map that can be applied whether you blog about math, science, technology, programming, or just about any other professional area.

Check it out and let me know what you think. If you’re just hearing about this book for the first time now, be sure to read the free introduction and excerpts to get an idea of how practical and useful this book is.

Happy technical and scientific blogging!

Outliers is the 2008 bestseller written by New Yorker magazine business and science writer Malcolm Gladwell, author of The Tipping Point, Blink, and What the Dog Saw.  Gladwell is the son of mathematician Graham Gladwell and Outliers has a lot to say about mathematics, both indirectly and directly in the chapters “Rice Paddies and Math Tests” and “Marita’s Bargain”.  Most importantly, Outliers has popularized the theory of deliberate practice espoused by psychologist K. Anders Ericsson.

Briefly, in the perennial “nature versus nurture” debate, Ericsson and, to a lesser extent Gladwell in his book, takes an extreme environmental position, attributing expertise and what is commonly referred to as “genius” or “talent” solely to many hours of “deliberate practice.”  Ericsson and Gladwell argue that success is due to performing at least ten-thousand hours of this deliberate practice, typically over ten years.  The title of chapter two in Outliers is The 10,000-Hour Rule.  Gladwell, in particular, uses the term deliberate practice vaguely in the book and his public presentations, often using it interchangeably with practice, as will be critiqued further below.  Ericsson is more precise in defining what he means by deliberate practice and distinguishing it from ordinary practice or experience.  This is important because there are obviously many examples of scientists, mathematicians, athletes, chess players, and others with far more than 10,000 hours of ordinary experience or practice in their field who do not perform at the genius or star level that Ericsson and Gladwell are talking about.

In Ericsson’s theory, deliberate practice involves many hours of heavy practice of relatively rare tasks or activities.  The example he has used on a number of occasions is the backhand in tennis.  The backhand is relatively rare.  Deliberate practice involves, amongst other things, practicing the backhand heavily to master this relatively rare move.  Consequently, a tennis player who has engaged in this sort of practice can, on average, easily beat a player who has mere experience or even less intensive forms of practice.  Now, in fact, Ericsson’s definition of deliberate practice is not quite this simple.  Sometimes he refers to deliberate practice as practice in which there is a strong conscious focus on self-criticism and self-improvement, intensively examining one’s performance for errors and room for improvement.  Deliberate practice is difficult to define across many different fields and activities.  What is deliberate practice in tennis may not be deliberate practice in mathematics.  What is deliberate practice in pure mathematics (theorem proving) may not be deliberate practice in applied mathematics (numerical computation, for example).

Outliers is largely composed of a series of stories that illustrate Gladwell’s main points.  Many of these stories are open to alternative interpretations and some are highly questionable.  This article will discuss a number of the stories in Outliers, pointing out their problems, with a special focus on deliberate practice and mathematics.

A Straw Man

Like many political works across the political spectrum, Outliers is guilty of setting up a straw man that is easy to debate and defeat.  Gladwell treats the reader to some seemingly absurd quotes from Florida Governor Jeb Bush, President George W. Bush’s brother and President George H.W. Bush’s son, describing himself as a “self-made man”.  In Outliers, Gladwell’s unnamed debate opponent, possibly Jeb Bush, is arguing that success is solely about individual talent and genius.  There are no significant environmental, cultural, or other factors involved.  Just because your Dad is the President has nothing to do with your relative success, say compared to that guy across town who’s Dad is a janitor.  Certainly not.  I did it all myself.  Of course, this is a position taken by very few, probably not even Jeb Bush despite the quotes.  Rather the actual debate is between those who view inherent talent, genius, hard work, and so forth as relatively more important on average than environmental factors, especially in “first world” nations such as the United States which ostensibly have democracy, free markets, and the rule of law.

Correlation is Not Causation

Outliers opens with the story of Roseto, an Italian-American community in Pennsylvania with a strikingly low rate of heart disease even though the residents of Roseto appear to share many of the allegedly unhealthy dietary habits of other Americans.  Gladwell recounts how medical researchers have theorized that this healthy state of affairs is due to the traditional family and social culture maintained by the Italian-Americans in Roseto, in which respect Roseto apparently does differ from neighboring communities and the United States as a whole.   This incidentally flies directly in the face of the medical orthodoxy blaming heart disease and heart attacks on diet, lack of exercise, fatty foods, and especially cholesterol levels which can be controlled by prescription drugs.

The problem is that what the researchers have demonstrated in Roseto is a correlation between an unusual lifestyle, for Americans, and better health.   That is all.  They have not proven this is the cause and the data would appear to directly contradict the causal theories of heart disease and heart attacks favored by medical researchers and pharmaceutical companies peddling anti-cholesterol drugs.

Indeed, this is a common problem in modern medicine and medical research.   Many reigning views — paradigms in the overused language of the philosopher of science Thomas Kuhn — are based, on close examination, on statistical correlations between disease or health and various factors.  Much of the modern theory of heart disease and heart attacks is based on the famous Framingham, Massachusetts heart disease study which showed correlations between various factors such as weight and diet and heart disease and heart attacks.  However, there are other studies such as the Roseto study touted by Gladwell that appear, at least to the naive observer unschooled int he elaborate theories that modern researchers frequently produce to explain otherwise grossly contradictory data, to disagree.

Thus, the Roseto story may illustrate the primacy of culture over other individualistic causes as Gladwell proposes.  Then again, it may not.  It may simply be another data point indicating that we don not understand heart disease and heart attacks nearly as well as the medical experts believe (or, at least, claim).

It is Not What You Know; It is When You Were Born

Gladwell follows with a story about champion Canadian hockey players, who remarkably are nearly all born in the months of January, February, and March of each year.  Shocking!  Very few would disagree with the proposition that success in hockey or other competitive activities should be about talent, skill, determination, and so forth, and certainly not which month you were born in.  This oddity is thought to be due to the annual age cutoff in hockey competitions in Canada which gives an advantage to kids born at the start of the year who are the oldest and thus largest and most mature members of their age cohort in each year.

Certainly this is unfair and a better system would compensate for this effect.  But, how much does it tell us about the relative importance of innate talent, if it exists, and environment?  Actually, it may tell us very little.  After all, the vast majority of kids in Canada who try out for hockey don’t end up in the champion teams.  What this oddity may tell us is only that, among those with the highest innate hockey talent, those born in the first few months of the year have an unfair advantage over their otherwise talented competitors.

If there is an effective cutoff in talent or size or some other variable innate to the person, then the very best will be sorted solely by external or random factors such as birthdates.  This does not tell us that environment or culture is King, but rather that there is an upper bound to innate talent and there are more competitors at the upper bound than available positions.  As the global population expands, we would expect more and more people to exist at the upper bounds of human potential.

As a specific example, suppose that innate hockey ability can be measured on a special scale from 0 to 100.  The potential best hockey players in the world score 100.  One-hundred thousand Canadian kids compete for 1000 positions on twenty national hockey teams.  If there are more than one thousand Canadian would-be hockey players who score 100 on the innate hockey ability scale, purely environmental or chance factors will start to kick in and contribute to the players who end up on the teams.  If there are 10,000 top hockey players competing for 1000 positions, the environmental effects are likely to be very strong.  For example, only players born in early months in the year who have an unfair advantage in size and maturity may be selected.  This does not mean innate ability is not very important relative to environment.  In fact, 90,000 players fail because they lack the innate ability.  Only 9,000 players fail to make the teams because of environmental effects.

Gladwell plays up the clustering of the birthdates of software entrepreneurs, corporate takeover lawyers, and others in several other places in the book.  The same objection noted above applies to these examples as well.

A Hard Day’s Night

Outliers then introduces the concept of deliberate practice and the 10,000 hour rule in earnest.  To illustrate Ericsson’s theory Gladwell tells the titillating tale of the Beatles sojourn in Hamburg, Germany where they ostensibly acquired ten-thousand hours of deliberate practice playing seven days a week, eight hours per day or more, at — yes — a strip club.  Gladwell returns to this tale several times in Outliers and obviously enjoyed telling the story during his promotional appearances for the book; videos of many of his presentations are available on the Web.

With all due respect to the genuine musical talents of John Lennon, Paul McCartney, George Harrison, and Ringo Starr, popular music is clearly about a lot more than talent and indeed occasionally no genuine talent at all seems to be required.  There have, for example, been scandals in which a performer could not even sing; the voice was provided by someone else, perhaps less photogenic and less sexy.  There are very few fat or ugly people among popular music stars  The manufacturing of pop stars and bands is so obvious that it is the subject of repeated satire in popular culture, movies, and so forth.  The slimy music executive who manufactures a popular musician or band is a stock character in movies and television.  More to the point, did playing songs at a strip club in Germany really qualify as the tedious deliberate practice in Ericsson’s theory?  This is rather doubtful, but it is a catchy story with plenty of sex.

Is Bill Gates a Champion Software Developer?

Outliers also tells the tale of Bill Gates, the Microsoft founder and former CEO, whom Gladwell apparently interviewed.  In particular, Gladwell tells the story of how Bill Gates supposedly acquired at least ten-thousand hours of deliberate practice on a time sharing computer as a kid in the 1960s.  Outliers attributes Gates and Microsoft’s success in no small measure to this practice and Gates skills as a software developer.

As with the Beatles, it is difficult to evaluate this story.  Did it really happen?  Did Bill Gates practice on the time sharing computer qualify as Ericsson’s vaguely defined deliberate practice?  Most importantly, is BIll Gates and Microsoft’s phenomenal success due to technical proficiency in software development?

The answer to this last question is probably not.  Much of Microsoft’ success can be traced to a shrewd business deal in which Bill Gates sold an operating system, now known as DOS, that he did not have and did not create, to IBM for the new IBM PC.  IBM then proceeded to make a series of colossal missteps over the next several years which culminated in sabotaging their OS/2 operating system and handing the PC software market to Microsoft, essentially an IBM supplier.  Bill Gates is a shrewd businessman and negotiator; of this, there is no doubt.  Was he a champion software developer: most probably not.

What is Deliberate Practice?

Both the story of the Beatles and the story of Bill Gates illustrate the problem of defining exactly what is deliberate practice.  In his book and public presentations, Gladwell usually sidesteps this issue, often using the terms deliberate practice, experience, and practice interchangeably.  Yet, the definition of deliberate practice is central to Ericsson’s theory.  There were and are thousands of chess players with many more hours of ordinary experience or practice than Bobby Fischer when he became an International Grand Master in chess.  The same with many other intellectual activities.  Yet, many highly experienced individuals do not perform at an extremely high level like Bobby Fischer in chess or Tiger Woods in golf or Albert Einstein in physics.  If just any experience or practice worked, most intellectual activities would be dominated by octogenarians.  Yet, in many activities this is not the case.  Why do some people with only 10,000 hours of experience grossly outperform others with 20 or 30,000 hours of experience?  It is this problem that requires the definition of a special kind of practice — deliberate practice — to avoid invoking innate talent or genius.

Chris and Oppie

In arguing for nurture over nature, Outliers tells the stories of Chris Langan, allegedly the smartest man in the world, and the famous physicist J. Robert Oppenheimer, the scientific director of the Manhattan Project which developed the atomic bomb during World War II.  Langan hailed from a poor, rather dysfunctional family and flamed out of a college allegedly due to administrative problems that led to the cancellation of his scholarship.  He has had a succession of rather unimpressive jobs, is now a farmer, and won $250,000 on the One versus One Hundred game show.  Oppenheimer was the brilliant son of a wealthy Jewish garment manufacturer in New York City and sailed through the attempted murder of his thesis adviser at Cambridge with a a slap on the wrist, to be later picked by General Leslie Groves as scientific director of the Manhattan Project — over many more highly qualified physicists such as Leo Szilard and Karl Compton, who do not seem to have had a track record of trying to murder someone.

Gladwell attributes Oppenheimer’s rather remarkable success, especially compared to Langan, to Oppenheimer’s sophisticated upper class social skills acquired during his posh upbringing — contrasted to Langan’s presumably poor working-class social skills.  This is somewhat curious.  By many accounts, Oppenheimer was quite arrogant and even admirers concede that his notorious arrogance contributed to his eventual downfall.  And, ahem, he had tried to murder his thesis adviser.

In Gladwell’s account, Oppenheimer talks his way out of an attempted murder charge.  There is not even a suggestion that Oppenheimer’s wealthy father pulled some strings (paid someone off) to make the potential attempted murder charges go away.  A poor man’s felony is a rich man’s prank.

Why would General Groves have selected Oppenheimer with his history of attempted murder, mental problems, extensive connections to the Communist Party through his wife, mistresses, graduate students, and friends, abrasive, arrogant personality, and limited resume over Leo Szilard, Karl Compton, or many other more qualified physicists of the time?  General Groves did not initiate the Manhattan Project.  By most accounts, Leo Szilard, a friend and business partner of Albert Einstein and a rather murky character in his own right, was the original mastermind of the program.  General Groves was brought in later as the project grew and quickly took a disliking to Szilard, forcing him out.  Groves was also embroiled in feuds with other prominent physicists such as Karl Compton.

In selecting Oppenheimer to run the Manhattan Project, Groves was promoting a relatively young (38), second tier physicist who would be heavily dependent on Groves patronage for his position: certainly not the case with Szilard or Compton, for example.  Oppenheimer’s dubious background probably ensured that Oppenheimer could be easily disposed of if he zigged when he was supposed to zag.  Years later, in disputes over the Air Force’s atomic bomber program and the hydrogen bomb, Oppenheimer appears to have zigged when he was supposed to zag and, indeed, he was easily discredited and removed from influence, becoming a martyr for the American Left in the process.

This blindness to politics is one of the recurring problems with Outliers.   In Gladwell’s account, the rich and powerful succeed because they have a better culture and because their parents take them to extra violin practice or pay for them to practice on time sharing computers.  It is not genes but hard work, deliberate practice, that Gladwell uses to explain success.  There is no politics, no string pulling, no skullduggery, no old boy networks or hidden agendas.

This real or feigned political naivete is particularly evident in Gladwell’s response to some of Chris Langan’s comments about Langan’s unhappy experiences with academia:

Even is his discussion of Harvard, it’s as if Langan has no conception of the culture and particulars of the institution that he is talking about.  (Langan speaking) When you accept a paycheck from these people, it is going to come down to what you want to do and what you feel is right versus what the man says you can do to receive another paycheck. What?  One of the main reasons college professors accept a lower paycheck than they could get in private industry is that university life gives them the freedom to do what they want to do and what they feel is right.  Langan has Harvard backwards.

Oh, really?  This argument that researchers and scholars in academia could make more money in industry and thus must be motivated by altruism and academic freedom is extremely common.  It is repeated in both popular science and by academics themselves.  It comes up in job interviews when academic researchers are recruiting people to work for their labs.  It plays a role in the decision by many to go to graduate school, pursue a Ph.D., and try to become a professor, often a disappointing experience.  It also gives the pronouncements of academics and researchers the added authority and credibility of the selfless truth seeker.  But is it true?

Researchers and scholars may pursue a career in academia because they believe they have a special aptitude, whether inborn or acquired, for their specific field.  I have talked with physicists who cited this as their reason for pursuing a career in physics.  If one believes that one has a high general intelligence and could do whatever one wants, then one might pursue a career in physics or another academic field out of altruism.  On the other hand if one believes one has a special advantage in a specific field, then one may pursue an academic career in preference to industry because one expects to do well specifically in that field, better than one could do in industry.  Despite very high intelligence, Chris Langan did not do well in industry, at least until his game show appearance.  Indeed, many of the stories that Gladwell recounts, which call into question the notion of general intelligence, undermine this argument.

The deliberate practice that Gladwell touts in his book is about practicing specific problems and techniques used in specific narrow fields whether these are chess or golf or the exotic techniques of particle physics.  Deliberate practice is not the general intelligence of Lewis Terman that Gladwell ridicules.  Someone who through happenstance or careful thought engages in huge amounts of deliberate practice to score well in tests in a specific academic field does not, in many cases, have easily transferable skills to industry.  Gladwell’s own argument about the the true nature of success actually significantly undercuts the argument that he makes in discussing Chris Langan’s comments on Harvard.

Modern research is not a side activity of professors who are principally paid to teach students.  It is often discussed in popular science and by academics themselves as if it were, but it is not.  Rather it is largely funded by gigantic government bureaucracies such as the Department of Energy (DOE), National Institutes of Health (NIH), National Aeronautics and Space Administration (NASA), National Science Foundation (NSF), the Defense Advanced Research Projects Agency (DARPA), and several others.  All told, the United States federal government spends about $100 billion on activities that are classified as research and development.  Major universities such as Harvard receive billions of dollars in grants and contracts from these agencies.  Faculty are recruited and rewarded most especially for bringing in grant money from the government.  Modern researchers usually list the dollar amount of grants or contracts that they have received on their curriculum vitae  The very top researchers in major fields such as molecular biology or high energy physics do fairly well financially; it is not clear they could have done better in industry.  This does not mean that the vast majority of graduate students, post-doctoral researchers, and junior faculty, most of whom will not get a tenured faculty position and will eventually leave their field for industry, are not accepting low salaries and could do better in industry.  In most cases, grad students and post-docs who are US Citizens, have a green card, or who otherwise can move freely from job to job could find a better paying job in industry.

In many respects, Gladwell’s account is not that different from the conservative Republicans like Jeb Bush that he is presumably debating.  Instead of attributing the success of the upper class to some mysterious genetic advantage, Gladwell repeatedly invokes virtuous hard work, deliberate practice, and special cultural advantages such as superior social skills.

In comparing Chris Langan and J. Robert Oppenheimer, Gladwell is tearing down a straw man.  Coming from a very wealthy family, Oppenheimer had enormous advantages in addition to genius.  On the other hand, although a genius, Langan came from an extremely poor and dysfunctional family environment.  If one argues that innate talent always wins out, then this comparison easily settles the argument: no it does not.   Gladwell compares two performers for whom the environmental differences are extreme.  Even those who believe innate ability is a very strong determinant, say contributing ninety percent of outcomes on average, might still expect to see dramatic differences in outcomes and performance between an Oppenheimer and a Chris Langan.  What happens when we compare a genius like Chris Langan from a stable, middle class family with someone of average intelligence also from a stable, middle class family?

The Problem with Deliberate Practice

In the United States (at least since Sputnik in 1957), math and science education are always in trouble.  The schools are always failing to teach math and science.  There is always a desperate shortage of mathematicians and scientists.  Ten Ph.D.’s for every tenure track position is way too few.  The United States desperately needs more Ph.D.’s.  Some other nation — Russia, Japan, and now China — is always running circles around the spoiled, wimpy, not to mention dumb American school kids.  The sky is falling! Send more money!  In the chapters “Rice Paddies and Math Tests” and “Marita’s Bargain” Gladwell discusses the perennial issue of math education in the United States and makes specific suggestions to improve math education based on the theory of deliberate practice as he sees it.  Perhaps not surprisingly, the bottom line is that students should work, study and practice harder.  Gladwell proposes to abolish summer vacation and replace it with many more weeks of drilling and drudgery in hopes of producing a nation of virtuous Asian-style math whizzes (China being the current foreign nation running circles around America’s mediocre math and science students).

There is another, less flattering name for deliberate practice: rote memorization.  Anders Ericsson’s research focuses heavily on activities that involve short, timed contests or exams: competitive games like chess, sports, music performance, and, of course, academic exams and tests.  If a test or competition involves activities that are too complex to figure out from first principles in the time allotted — even for a genius — then one must practice and memorize the activity to succeed in the test or competition.  A tennis player does not have time to figure out how to hit a backhand during a tennis match.  He or she had better know how to do it to defeat a player who does.  Only the most extreme innate intelligence, like the fictional hero of the Pretender TV show, could adapt and perform in real time.

There is no doubt that there are many real world, relatively repetitive activities that require practicing and memorization.  We rightly expect air traffic controllers and pilots to know what to do, how to fly an airplane, and so forth.  We do not want them to be figuring out what to do while flying a plane with hundreds of passengers.  But does memorizing the answer to a problem that we (humankind) already know how to answer teach the student how to solve a problem that we don’t know the answer to: curing cancer, free energy,  an end to war?

Invention and discovery, research and development, software development, and many other activities, including, for example, responding to a new kind of emergency, depend on the ability of scientists, engineers, and ordinary citizens to figure out new things and improvise.  Deliberate practice will not necessarily teach these important skills; in many cases, we do not even know what these skills are.

Gladwell quotes several famous education reformers from the nineteenth century with evident disdain for their enthusiasm for summer vacations and (OH MY GOD) play:

It is when thus relieved from the state of tension belonging to actual study that boys and girls, as well as men and women, acquire the habit of thought and reflection, and of forming their own conclusions independently of what they are taught and the authority of others.

Gladwell expresses astonishment at the nineteenth century notion that one could study too much or much or that it could have adverse consequences such as insanity, attributing such mushy thinking to the wimpy Western farming tradition as opposed to the virtuous work-until-you-drop tradition of the Asian rice paddy.

Yet, there is a striking pattern in many major invention and discoveries that the nineteenth century education reformers may have been well aware of.  A very high proportion of major inventions and discoveries have been made on a break, a vacation for example, when not actively thinking about the problem.   According to Greek historical accounts, Archimedes solved the problem of determining the gold content of the King’s crown without destroying the crown (a major breakthrough) while taking a bath.  Kepler realized that Mars had an elliptical orbit over the Easter Holiday in 1605.  The mathematician William Rowan Hamilton conceived of quaternions during a recreational walk.  In his autobiography, Nikola Tesla described suddenly seeing the design of the alternating current motor in his head while watching the sunset and discussing Goethe.  James Watt claimed he conceived of the separate condenser steam engine during a walk in the park.  Erwin Schrodinger came up with the Schrodinger Equation on a ski vacation in the Alps.  In his book on mathematical invention, the great French mathematician Jacques Hadamard concluded that this was part of the general pattern of inventions; they frequently occurred during a break.

There is also evidence in support of the nineteenth century notion that one can overdo study.  The famous mathematician Felix Klein burned himself out in his scholarly duel with Poincare and lost his ability to perform many mathematical research activities, devoting himself to administration and mentoring other mathematicians.   Several famous mathematicians including Georg Cantor and Kurt Godel developed serious psychological problems.  Godel may have starved himself to death.  Many inventors and discoverers have described a period of mental exhaustion after making their “breakthrough”.  Modern software developers often experience “burnout” after prolonged programming projects.  Software development has very high turnover rates and many people enter and, more to the point here, leave the field every year.

Deliberate practice presents the hazard of substituting rote memorization for deeper understanding.  In many respects a form of studying for the exam, it likely can deliver high levels of performance on tests and exams, substituting easily measurable technical skills (such as basic arithmetic) for harder to measure abstract reasoning skills and “intuition” that are essential to solve many problems that we do not yet know how to solve.

© 2011 John F. McGowan

About the Author

John F. McGowan, Ph.D. is a software developer, research scientist, and consultant. He works primarily in the area of complex algorithms that embody advanced mathematical and logical concepts, including speech recognition and video compression technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

We’d also like to learn a bit more about the demographic that comprises our readership. For this reason, we’d like to kindly ask you to take our survey. It shouldn’t take you more than 5 minutes to fill out this questionnaire. We’ll also publish the results for non-open ended questions, in aggregated form (so that none of your information will be identifiable). Thank you very much for your time and support of Math Blog.

Take our survey

LaTeX

The word processing program has a visual equation editor that works much like Microsoft Equation Editor. This is not a bad option for someone who only occasionally needs to include equations in their writing. However, professionals who write extensively on mathematical topics will be better served by investing the effort in learning to use LaTeX for typing equations.

The new equation editor in Google Docs will recognize LaTeX symbols and will automatically convert them to the appropriate symbol. So, for instance, you can type \alpha and get .

This has the potential to be quite beneficial for groups that need to collaborate on mathematical documents since it leverages the LaTeX shortcuts for users who are already familiar as well as providing a graphical tool for users who prefer that option.

If you’re interested in using these shortcuts, a listing of common LaTeX symbols for mathematics is available here.

Connected Drawings

The other big feature is that elements in a drawing can be connected in such a way that the connection will stretch when the elements are moved. Google provides a detailed explanation here along with this sample image.

This type of drawing is useful in mathematics (particularly graph theory), computer science and engineering. I have used the open source GraphViz in the past for drawing connected graphs. While this new feature won’t replace a program like GraphViz, it is a feature that will certainly be useful for scientific writers.

Conclusion

Although I wasn’t a big fan of web apps at first, Google Docs won me over by providing a minimal set of tools and amazingly simple collaboration. I’m very glad to see that they are adding features that will appeal to technical writers.

About

Tony McDaniel is a graduate student in computational engineering at the University of Tennessee at Chattanooga. His research interests include computational mathematics, algorithm design and analysis, and data visualization for numerical solutions of partial differential equations. Other interests include photography, model rocketry and electronics.

Sponsor’s message: Check out Math Better Explained, an insightful ebook and screencast series that will help you see math in a new light and experience more of those awesome “aha!” moments when ideas suddenly click.

(Tip of the hat to Flowing Data & Revolutions)

Prediction is very difficult, especially about the future.

Common quotation, variously attributed to Yogi Berra, Niels Bohr, Mark Twain and others.

Introduction

In Isaac Asimov’s famous science fiction novel Foundation, a group of scientists in the distant future led by Hari Seldon discover a mathematical method to predict the course of future events, anticipating the collapse of the reigning Galactic Empire into a new Dark Age. Armed with the mathematical methods of Hari Seldon’s “psychohistory,” the scientists create a Foundation on the edge of the Galaxy that saves civilization from the prophesied Dark Age. The notion that mathematics can be used to predict human behavior in economics, finance, politics, and many other fields and activities has great appeal. With the spread of increasingly powerful computers, complex mathematical models of economics, finance, and other human activities have become more and more common. Often, the motives are much less admirable than the selfless super scientists of Asimov’s tale. Often, too, the accuracy and performance of the mathematical models has been much less impressive than Asimov’s fictional new science of psychohistory.

In the last few years, complex models of the value of mortgage backed securities have proven disastrously incorrect, a major contributing factor to the Great Recession, the present financial crisis. This is only the latest in a succession of such failures in quantitative finance. Similarly, many sophisticated econometric models of the economy have proven unreliable. To the extent that these models are shaping public policy, personal and corporate investment decisions, and so forth, the pitfalls of mathematical modeling and seemingly abstruse issues in the philosophy of science such as Karl Popper‘s doctrine of falsifiability are having a substantial impact on people and society.

In fact, applying mathematical models to economics, finance, and other human activities is especially treacherous. All mathematical modeling suffers from the deep problem that one can construct an infinite range of functions that approximate current observations and data arbitrarily well and yet make any and all possible predictions about possible new observations. In practice, human beings use various criterion to select mathematical models that are likely to be true, many of which criterion cannot be justified in any rigorous or rational way and some of which criterion are difficult to identify (intuition, “that just doesn’t feel right”, “God doesn’t play dice with the universe.”). Nonetheless, human judgment has a high error rate, though surely much more accurate than a blind guess.

In addition to this pervasive problem, many of the assumptions used in applying mathematical models to physical processes such as the motion of the planets or radioactive decay surely do not apply to economics, finance, or other forms of human behavior. Our expectation is that the motion of the planets is governed by the same “laws” today as last year or last decade or in 1605 when Johannes Kepler first recognized the elliptical orbits of the planets. Indeed, this regularity of many natural phenomenon is strongly born out by experience. On the other hand, the economy or financial markets or other human activities change and evolve. However imperfectly, human beings learn from mistakes, develop new technologies and processes. Human beings are herd animals and prone to fads and fashions that have no parallel in physical phenomena. One would not expect the market for gold to be the same today as in 1960 and it was not. The price of gold was fixed by the US federal government in 1960 and today it is not.

This article takes a look at the price of gold since 1970 when the United States ended the gold standard. Gold is currently rising sharply in price as it did in the late 1970′s and early 1980′s. This article will show how it is possible to construct many different purely symbolic mathematical models of the price of gold that make different predictions. Indeed, one can cook up whatever prediction one would like. The article will also discuss the serious problems with applying the mathematical methods of physics and other “hard” sciences to the price of gold as a specific example of a general problem.

Mathematics for Goldfinger

With the collapse of the Bretton-Woods foreign exchange system (1968-1971), the price of gold, previously fixed to the US dollar, was allowed to float free. Since 1970 the price of an ounce of gold in US dollars has risen substantially.

Nominal Gold Price

The United States government reports a consumer price index (CPI) that is purported to reflect the cost of living in US dollars for a typical US citizen. Although there are some reasons to be skeptical about the accuracy of the CPI in recent years, this article will use the CPI as a proxy for the overall price level in the United States.

United States Consumer Price Index

One can see that the CPI has generally risen at a higher rate since the United States ended the gold standard in 1970. This also corresponds to a period of overall rising energy prices and very limited progress in power and propulsion technologies. Many other areas have seen relatively limited scientific and technological progress during this period. Computers and electronics have, of course, continued their historical trend of rapid progress to the present.

Historically, the US inflation rate was high during the periods of World War I and World War II, but otherwise generally lower than the present period, since 1970. The official inflation rate, the rate of increase of the CPI, was especially high during the 1970s, a period of sharp increases in nominal and real energy prices. The inflation rate derived from the CPI in recent years does not seem to reflect the common experience of rising energy prices since the late 1990s. Nor does it show any evidence of the sharp rise in housing prices in the US from 2002 to 2005; by many estimates, housing prices in urban areas remain high compared to the historical trend.

Investors are generally interested in the real inflation-adjusted value of an investment. The value of gold in real terms fluctuated somewhat during the period of the gold standard but has fluctuated much more since 1970. Particularly notable is the large spike in the real and nominal price of gold in the early 1980′s. In real terms, the current rise in gold price has not reached the level of the spike in the early 1980′s. If inflation has been higher than the official CPI, the current real price of gold would be even lower than the spike in the early 1980s.

One can construct a simple symbolic mathematical model of the real price of gold since 1970 (the gold standard disintegrated between 1968 and August 1971 when the US government ended any attempt to tie the dollar to gold) by using a polynomial with several terms:

Where is the gold price as a function of the time . This is a simple mathematical model with little real-world justification. It will, in fact, make grossly unrealistic predictions that should call it into question. A polynomial model of the time series of real gold prices is:

Polynomial Model of Gold Price (12 Terms)

One can approximate any continuous function as accurately as one would like with a sum of a series of powers. The problem is that a power such as will grow without bound as the independent variable grows. This is usually quite unrealistic. This is a simple illustration of the difference between a symbolic mathematical model of reality and our common sense every day sense of reality.

Apocalypse or Gold Bubble?

Many other mathematical models are possible. In fact, there are an infinite number of functions or curves that agree with the data during the period from 1970 to 2009. In this way, one can predict anything. In mathematical models of physical phenomenon, it is common to try to construct the model from a set of building block functions, or more generally terms such as terms in a differential equation. This may be arbitrary or justified by arguing that the building blocks represent fundamental building blocks of the physical process in some way. In practice, one is trying to capture regularities in the data that may recur in new data. For example, the position of a pendulum is periodic; it repeats over and over again. Simple harmonic motion of this type was one of the first kinds of behavior understood mathematically by the ancients. One can construct models of the gold price data that agree with the data quite well by eye using a collection of peaks rather than periodic functions. One might, for example, represent the seeming peaks in the gold price data as the Gaussian or Normal function, commonly referred to as the “Bell Curve,”

A very simple model is the sum of two Gaussians:

Gold 2 Gaussian Model

This simple model with two Gaussians does not agree very well with the data. It predicts the following future performance:

Gold 2 Gaussian Prediction

It predicts that the real price of gold will peak in about 2025 and then drop. One can get much better agreement between the model and the gold price data by adding more Gaussians, loosely corresponding to the apparent gold peaks in about 1974, 1982, 1987, 1993, and today. The spike in 1982 is especially sharp and can better be approximated by combining two Gaussians.

Gold 7 Gaussian Model

Now the agreement by eye is much better. The curves appear essentially the same. This model makes a different prediction.

Gold 7 Gaussian Prediction

This model predicts a peak in real gold prices in just a few years, about 2012, followed by a decline to essentially zero. One can get significantly different predictions simply by using a different function to model the peaks in the gold price. For example, one can use the Cauchy-Lorentz distribution as a model for the peaks:

where is the time (year in this case), is the location of the peak, and is a measure of the width or dispersion of the peak. Initially, one can try a model with two peaks:

Gold 2 Cauchy Model

This model looks very similar to the two Gaussian model. It predicts something different however.

Gold 2 Cauchy Prediction

In this case, the price of gold trails off slowly instead of dropping to essentially zero in about a decade. This is a difference between the Gaussian and the Cauchy-Lorentz functions. Of course, the agreement between the model and data is not very good. One probably would not and should not trust it. One can achieve better agreement with more peaks, just as in the Gaussian models.

Gold 7 Cauchy Model

Now the agreement is almost exact by eye. The prediction however differs from the seven Gaussian model.

Gold 7 Cauchy Prediction

Again, the price of gold trails off slowly over a period of decades due to the difference between the Gaussian and Cauchy-Lorentz peak models. In fact, one can get essentially any prediction by choosing the appropriate mathematical model. How reasonable are these predictions? There are several theories about the present rise in gold prices. One clear contender is that the gold price rise in the last decade is yet another speculative financial bubble. In this scenario, one would expect the price of gold to drop substantially within a decade. One might also argue that the sharp run-up in gold prices will encourage overproduction of gold and the development of better gold refining, recycling technologies, alternatives to gold, and even perhaps the alchemist’s dream of converting base metals into gold (using nuclear reactors for example). At the other extreme, the rise in gold prices is tied to apocalyptic scenarios in which the US and other governments go bankrupt due to deficit spending and the dollar and other paper currencies becomes as valueless as the German mark during the infamous German hyperinflation. Thus, the real inflation-adjusted price of gold spikes as investors seek an inflation proof haven. Unfortunately, civilization collapses. Gold and other luxury items become valueless in the bitter Darwinian battle for survival in the post-apocalyptic world. The most valuable possessions are a gun, ammunition, and a large horde of dried food — all purchased from advertisers on Alex Jones web site. In either scenario, gold eventually tanks.

All kidding aside, it is usually possible to find technically sophisticated, plausible justifications for mathematical models. Usually does not mean always. The behavior of the polynomial models is grossly unrealistic. How likely is it that the price of gold would go negative, let alone extremely negative — meaning that people would be paying large sums of money to get rid of gold? The evidence that the Earth is nearly spherical with a diameter of about 8000 miles is very strong. Any argument that the Earth is really flat is extremely difficult to make in the present day. Most of us are pretty confident that the Sun will set today and rise again tomorrow and continue to do so for many, many years to come. There is some knowledge that is pretty certain. There are some mathematical models that have proven extremely accurate and reliable.

Nonetheless, it is usually possible to find plausible arguments for mathematical models and conceptual theories. One can usually explain away even grossly contradictory — “falsifying” in the language of Karl Popper — observations or experiments in a technically sophisticated, plausible way. This can create the illusion that a theory or mathematical model falls into the category of almost certain knowledge, our common sense notion of a fact. Once upon a time, many people believed that it was a fact that the Earth was flat. How could it possibly be a sphere? Political power, social status, and sizable funding tend to flow to those who claim certainty, to know hard facts rather than speculative theories. In common thinking, an expert is someone who says “I know the answer.” not someone who says “I don’t know.”

Scientists and science popularizers often seek to promote reigning scientific theories to the level of “almost certain” knowledge or “facts,” such as that the Earth is roughly spherical or the historical fact of the Holocaust. These latter two are the most common analogies cited in the popular science literature. Scientific theories are said to be supported by “overwhelming evidence.” There is a “consensus” that the theory is correct. The theory is no longer a theory, but a fact. In debates about evolution and creation, one encounters the curious claim that evolution is a falsifiable theory, that creationism or “intelligent design” is not falsifiable and thus not scientific, and that evolution is also a fact beyond any reasonable dispute or falsification. Increasingly, malcontents or die-hards who question the “consensus” are “deniers” or “denialists” in analogy to Holocaust deniers, an extremely emotional analogy indeed. There are now evolution deniers, AIDS deniers, global warming deniers, and a growing list of other deniers. Yet, certain knowledge is relatively rare especially in the frontiers of science.

In the case of gold, one can make a strong argument that the models above are unlikely to be correct. There is a certain fundamental demand for gold for industrial uses, in electronics for example. There is a long history of people buying gold for jewelry. The economic and political problems, the fears and the greed, that have historically driven the fluctuations in the price of gold are likely to continue through the 21st century. Thus, one should doubt models that show the real price of gold dropping to zero or nearly zero. Yet, both symbolic mathematics and verbal reasoning enable one to make a plausible argument that this will happen, supported by fancy symbolic mathematics and technical graphs.

A Deep Problem

It is always possible to perfectly match or fit any set of data points using a linear combination of at least linearly independent functions. Under many conditions, a function or composition of building block functions with adjustable parameters can also match or fit any set of data points. In addition, with creativity and trial and error, one can often construct mathematical models with less than parameters or building block functions that nonetheless do a good job of matching or fitting data points. These models may often fail to predict future data or data that lies outside of the region used to make the model or fit.

In general, a composition of a set of building block functions has a sort of plasticity, a certain ability to match any data to some degree. If there are building block functions or adjustable parameters and data points, this plasticity can be rigorously shown to be essentially infinite, that is the model can always match any set of data points. The philosopher of science Karl Popper called such models or theories unfalsifiable. They can never be proven wrong. They predict everything and therefore they also predict nothing. Unlike popular presentations of his doctrine of falsifiability, Popper recognized that falsifiability was often not a black and white distinction. There were degrees of falsifiability and he tried to define some logical and quantitative criterion for the degree of falsifiability. What this means is that a mathematical model or theory with adjustable parameters where is less than the number of data points may still match a broad class of possible observations or data. It can be falsified, perhaps, but only with great difficulty. There is, in the examples above, a lot of freedom to match many different sets of data with seven peaks in the model. The models have 21 adjustable parameters and there are 40 data points (40 years). Although the models cannot exactly match all sets of 40 data points, nonetheless they are likely to be good enough for most purposes. Remaining differences between the data and the models can easily be attributed to noise, measurement errors, or some similar excuse.

In physics, there are some examples of very large data sets that match very simple mathematical formulas. The motions of the planets in the Solar System conform to Kepler’s Laws and Newton’s theory of gravitation to a very high degree. This is not necessarily true for the motion of stars in the Milky Way galaxy, galaxies in galactic clusters and so forth. A deviation from Newtonian gravity is a possible explanation for the anomalies often cited as evidence for so-called “dark matter” or “dark energy.” So too, there is an enormous amount of quantitative data on the vibration of springs and strings, the motion of pendulums, simple radioactive decays, and various other physical phenomena that show precise matches to simple mathematical expressions such as Hooke’s Law for springs or exponential decay for radioactive decay. It is fair to say that the amount of data points is now in the millions or more and the models have only a few adjustable parameters such as the half-life of a radioactive isotope. Consequently, it is probably reasonable to have a high degree of confidence in these models. At the other extreme, it is clear that we should put no confidence in the predictive power of models with as many adjustable parameters or building block functions as the number of data points.

In many situations however including the frontiers of human knowledge and often areas like economics, the situation falls into a gray area. The theories, both symbolic mathematical models and conceptual models, may be complex, but not so complex as to be obviously “unfalsifiable.” The amount of data may be limited or of questionable quality, but still not obviously inadequate to draw conclusions. It is here that faulty human judgment is in practice applied, in part because human judgment, fallible though it undoubtedly is, is still the best option, more reliable than symbolic mathematics or computer programs in many cases. In constructing mathematical models in these cases, human judgment is often hidden in the definition of the symbols and the choice of building block functions or other mathematical components used in the model.

Economics, finance, politics and other human activities are especially treacherous areas for mathematical modeling. The mathematical and scientific methods used in physics and other “hard” sciences were developed to study highly repeatable phenomena that do not appear to change appreciably over time. For example, a “fair” coin will come up heads when flipped on average half the time, tails half the time. This was true in the early days of probability and statistics during the Renaissance. It is true today. Fair games of chance work the same way today as yesterday, last year, last century, and presumably thousands of years ago or in the future. One can collect vast amounts of data on these games. Similarly, physical phenomena such as the motion of the planets, radioactive decay, and so forth appear to behave the same way time after time. They do not evolve, learn from mistakes, forget lessons learned, or suddenly change for difficult to understand reasons. None of this is true of much economic or financial data.

In the example of gold, the behavior of gold changed radically from 1968 to August 1971 when the gold standard ended. Since 1970, there have been extensive political, economic, and technological changes that probably effect the price of gold. The Cold War ended. Apartheid in gold producing South Africa ended, followed by a large exodus of the white minority who dominated the mining industry. The use of electronics has increased; gold is used in electronics. Women’s fashions in clothing and jewelry have changed. The terrorist attacks of September 11, 2001 probably contributed to general unease and the rise in gold prices. Against these many changes, there is only forty years of data on gold prices. Day to day gold prices are correlated and gold seems to follow trends over several years, seemingly rising and falling in peaks. Thus, there is very limited data to analyze compared to a physical process. At the same time, one still has the freedom to construct many different mathematical models, theoretically an infinite number.

This ability to construct many different models that agree with observations or experiment but which make quite different predictions is not a problem unique to symbolic mathematical models. Rather, we encounter the same or a similar problem in everyday life, in politics, in personal relationships, where concepts, words, and pictures are the norm rather than symbolic mathematics with its illusion of certainty. It occurs when opposing attorneys are able to present radically different interpretations of the same evidence in a court case. It occurs when political activists explain the same events in radically different ways that almost always confirm their beliefs. It occurs in disputes between co-workers where each sees the same event or problem quite differently (it is all your fault). It occurs in conflicts between husbands and wives when each sees the same events differently. If verbal concepts and mental pictures are actually mathematical models maintained in the brain (but not in a symbolic way), then it may well be exactly the same problem as that encountered in mathematical modeling.

There is no doubt that human judgment is faulty and limited. In some relatively rare cases mathematics or formal logic can clearly outperform human judgment. Nonetheless, in many situations, human judgment and intuition still win out over mathematics, formal logic, or computer programs. It remains an unsolved and perhaps unsolvable problem to find a way to select the right model that, in fact, predicts new observations as well or even better than human judgment.

The nature and origin of human judgment and intuition remains an enigma. Governments have spent billions of dollars and decades on artificial intelligence in the mostly futile effort to replicate even sometimes seemingly simple aspects of human reasoning. Human beings often cannot explain either verbally or in symbolic logical or mathematical ways their successful reasoning processes. Ancient scholars and philosophers might attribute their ideas to divine inspiration or mystical insight. Indeed historical accounts of inventions and discoveries are replete with reported sudden insights or realizations such as the famous story of Archimedes in his bathtub suddenly realizing how to determine the gold content of the King’s crown without destroying the crown and then racing naked through the streets of Syracuse shouting “Heureka!” One may wonder what Archimedes feared the King would do to him if he had failed to solve the problem. The modern scientific view would probably attribute this ability of the human mind to find the right answer to an anthropic or evolutionary cause. Our brain incorporates billions of years of evolution and is thus tuned to the mysterious underlying logical or mathematical structure of the universe.

Conclusion

It is important to realize that one can construct an infinite number of mathematical models that match a set of data. In some sense, one can construct an even “larger” number of mathematical models that match a set of data “well enough,” where remaining differences can easily be attributed to noise, instrument error, or minor effects that can be ignored for practical purposes. In high school and college, one is often exposed to mathematics and geometry as a rigorous deductive system. The epitome of this is Euclid’s geometry which many people are exposed to in high school; high school math courses typically teach the first three of Euclid’s thirteen books. One starts with axioms and definitions that often seem self-evident, with the possible exception of the so-called parallel postulate. One can apply a sequence of logical steps to get a precise unambiguous answer. Similarly, high school and college science courses frequently focus primarily on extremely well-measured phenomena such as vibrating springs or radioactive decay that precisely follow simple mathematical laws. Scientists are often described as “deriving,” “figuring out,” or “deducing” mathematical laws such as Newton’s Theory of Gravitation, Maxwell’s Equations of Electromagnetism, or Schrodinger’s Equation for quantum mechanics. The implication is that these mathematical theories can be found by the application of rigorous mathematical or logical rules, much in the way that theorems in Euclidean geometry are proven. This has a great appeal compared to messy, fallible human judgment and intuition. But the reality is that the theories were found through the application of messy, fallible human judgment and intuition, perhaps assisted by some mathematics, by model fitting methods, and so forth, but in the end it was mysterious human judgment and intuition.

If we could understand what human beings are actually doing, this would be a great advance. It would be an even greater advance to find a way to improve human judgment and intuition, which is certainly quite fallible.

In the meantime, it is particularly hazardous to try to apply mathematical modeling to economics, finance, and human behavior. This does not mean that we should not try. Nor does it mean that there may not be successes in applying mathematics to human activity. Indeed, in economics, there are some mathematical rules of thumb that are often correct. It is, for example, generally observed that inflation is lower when unemployment is higher; this relation loosely follows a mathematical curve. However, we are a very long distance from a predictive mathematical method comparable to Isaac Asimov’s fictional psychohistory if this is even possible. Most people don’t think in mathematics; can human behavior really be reduced to mathematics?

The present financial crisis illustrates that these seemingly abstruse issues of mathematical models can impact the lives of many people and organizations. This is, if anything, likely to increase with growing reliance on computers and mathematical models implemented in computer software and hardware. There remains no greater wisdom than the ancient Latin saying: Caveat Emptor (Buyer Beware).

Note: An appendix with the technical details of the analysis and plots presented above follows the Suggested Reading/References section below. This includes the raw data, the models, and the Octave scripts used to fit the models to the annual gold price data. This article is primarily for informational and educational purposes. It is not investment advice.

Copyright © 2010 John F. McGowan, Ph.D.

About the Author

John F. McGowan, Ph.D. is a software developer, research scientist, and consultant. He works primarily in the area of complex algorithms that embody advanced mathematical and logical concepts, including speech recognition and video compression technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Suggested Reading/References

Karl Popper, The Logic of Scientific Discovery, Routledge, London, England 2000 (First published 1959, English translation with new notes and appendices of Logik der Forschung, published Vienna, Austria, 1934)

Paul Feyerabend, Against Method (3rd Edition), Verso, 1993

Thomas Kuhn, The Structure of Scientific Revolutions, University Of Chicago Press; 3rd edition (December 15, 1996)

John D. Barrow and Frank J. Tipler, The Anthropic Cosmological Principle, Oxford University Press, New York, 1986

Isaac Asimov, Foundation, Spectra; Revised edition (October 1, 1991)

Roger Lowenstein, When Genius Failed: The Rise and Fall of Long Term Capital Management, Random House, New York, 2000

Emanuel Derman, My Life as a Quant: Reflections on Physics and Finance, John Wiley and Sons, Hoboken, New Jersey, 2004

Charles MacKay, Extraordinary Popular Delusions and the Madness of Crowds, Farrar, Straus, and Giroux, New York, 1932 (first published in London in 1841)

Robert J. Shiller, Irrational Exuberance, Broadway Books, New York, 2000

Dean Baker, False Profits: Recovering from the Bubble Economy, Polipoint Press (January 15, 2010)

Michael Specter, Denialism: how irrational thinking hinders scientific progress, harms the planet, and threatens our lives, Penguin Press, 2009

Seth C. Kalichman, Denying AIDS: Conspiracy Theories, Pseudoscience, and Human Tragedy, Springer, 2009

Bill McKibben, “Hot Mess: Why are conservatives so radical about climate?”, The New Republic, October 6, 2010

Appendix: Technical Details

The annual gold price data used in this article is from the World Gold Council. Here is the actual raw data.

1900    20.67    4.25        o    1900    8.14    0.037941643    544.78            1913-01-01    0.10
1901    20.67    4.25        o    1901    8.24    0.038407756    538.17            1913-01-02    0.10
1902    20.67    4.25        o    1902    8.34    0.03887387    531.72            1913-01-03    0.10
1903    20.67    4.25        o    1903    8.53    0.039759485    519.88            1913-01-04    0.10
1904    20.67    4.25        o    1904    8.63    0.040225599    513.85            1913-01-05    0.10
1905    20.67    4.25        o    1905    8.53    0.039759485    519.88            1913-01-06    0.10
1906    20.67    4.25        o    1906    8.72    0.040645101    508.55            1913-01-07    0.10
1907    20.67    4.25        o    1907    9.11    0.042462944    486.78            1913-01-08    0.10
1908    20.67    4.25        o    1908    8.92    0.041577328    497.15            1913-01-09    0.10
1909    20.67    4.25        o    1909    8.82    0.041111215    502.78            1913-01-10    0.10
1910    20.67    4.25        o    1910    9.21    0.042929058    481.49            1913-01-11    0.10
1911    20.67    4.25        o    1911    9.21    0.042929058    481.49            1913-01-12    0.10
1912    20.67    4.25        o    1912    9.4    0.043814673    471.76            1914-01-01    0.10
1913    20.67    4.25        12/31/1913    1913    9.6    0.04630723    446.37    10        1914-01-02    0.10
1914    20.67    4.25        12/31/1914    1914    9.69    0.046770302    441.95    10.1        1914-01-03    0.10
1915    20.67    4.25        12/31/1915    1915    9.74    0.047696447    433.37    10.3        1914-01-04    0.10
1916    20.67    4.25        12/31/1916    1916    10.64    0.053716387    384.80    11.6        1914-01-05    0.10
1917    20.67    4.25        12/31/1917    1917    12.82    0.063440905    325.82    13.7        1914-01-06    0.10
1918    20.67    4.25        12/31/1918    1918    15.06    0.076406929    270.53    16.5        1914-01-07    0.10
1919    20.67    4.50        12/31/1919    1919    17.3    0.087520665    236.17    18.9        1914-01-08    0.10
1920    20.67    5.60        12/31/1920    1920    20.04    0.089836026    230.09    19.4        1914-01-09    0.10
1921    20.67    5.35        12/31/1921    1921    17.9    0.080111508    258.02    17.3        1914-01-10    0.10
1922    20.67    4.69        12/31/1922    1922    16.77    0.078259219    264.12    16.9        1914-01-11    0.10
1923    20.67    4.51        12/31/1923    1923    17.07    0.080111508    258.02    17.3        1914-01-12    0.10
1924    20.67    4.68        12/31/1924    1924    17.1    0.080111508    258.02    17.3        1915-01-01    0.10
1925    20.67    4.25        12/31/1925    1925    17.53    0.082889942    249.37    17.9        1915-01-02    0.10
1926    20.67    4.25        12/31/1926    1926    17.7    0.081963797    252.18    17.7        1915-01-03    0.10
1927    20.67    4.25        12/31/1927    1927    17.37    0.080111508    258.02    17.3        1915-01-04    0.10
1928    20.67    4.25        12/31/1928    1928    17.13    0.079185363    261.03    17.1        1915-01-05    0.10
1929    20.67    4.25        12/31/1929    1929    17.13    0.079648436    259.52    17.2        1915-01-06    0.10
1930    20.67    4.25        12/31/1930    1930    16.7    0.07455464    277.25    16.1        1915-01-07    0.10
1931    20.67    4.25        12/31/1931    1931    15.23    0.067608556    305.73    14.6        1915-01-08    0.10
1932    20.67    5.90        12/31/1932    1932    13.66    0.060662471    340.74    13.1        1915-01-09    0.10
1933    20.67    6.24        12/31/1933    1933    12.96    0.061125544    338.16    13.2        1915-01-10    0.10
1934    35.00    6.88        12/31/1934    1934    13.39    0.062051688    564.05    13.4        1915-01-11    0.10
1935    35.00    7.11        12/31/1935    1935    13.73    0.063903977    547.70    13.8        1915-01-12    0.10
1936    35.00    7.02        12/31/1936    1936    13.86    0.064830122    539.87    14        1916-01-01    0.10
1937    35.00    7.04        12/31/1937    1937    14.36    0.066682411    524.88    14.4        1916-01-02    0.10
1938    35.00    7.13        12/31/1938    1938    14.09    0.064830122    539.87    14        1916-01-03    0.11
1939    35.00    7.72        12/31/1939    1939    13.89    0.064830122    539.87    14        1916-01-04    0.11
1940    35.00    8.40        12/31/1940    1940    14.03    0.065293194    536.04    14.1        1916-01-05    0.11
1941    35.00    8.40        12/31/1941    1941    14.73    0.071776206    487.63    15.5        1916-01-06    0.11
1942    35.00    8.40        12/31/1942    1942    16.3    0.078259219    447.23    16.9        1916-01-07    0.11
1943    35.00    8.40        12/31/1943    1943    17.3    0.08057458    434.38    17.4        1916-01-08    0.11
1944    35.00    8.40        12/31/1944    1944    17.6    0.082426869    424.62    17.8        1916-01-09    0.11
1945    35.00    8.61        12/31/1945    1945    18    0.084279159    415.29    18.2        1916-01-10    0.11
1946    35.00    8.61        12/31/1946    1946    19.54    0.099560544    351.54    21.5        1916-01-11    0.12
1947    35.00    8.61        12/31/1947    1947    22.34    0.108358918    323.00    23.4        1916-01-12    0.12
1948    35.00    8.68        12/31/1948    1948    24.08    0.111600424    313.62    24.1        1917-01-01    0.12
1949    35.00    9.40        12/31/1949    1949    23.85    0.109285063    320.26    23.6        1917-01-02    0.12
1950    35.00    12.50        12/31/1950    1950    24.08    0.115768075    302.33    25        1917-01-03    0.12
1951    35.00    12.50        12/31/1951    1951    25.98    0.122714159    285.22    26.5        1917-01-04    0.13
1952    35.00    12.50        12/31/1952    1952    26.55    0.123640304    283.08    26.7        1917-01-05    0.13
1953    35.00    12.50        12/31/1953    1953    26.75    0.124566449    280.97    26.9        1917-01-06    0.13
1954    35.00    12.50        12/31/1954    1954    26.88    0.123640304    283.08    26.7        1917-01-07    0.13
1955    35.00    12.50        12/31/1955    1955    26.78    0.124103376    282.02    26.8        1917-01-08    0.13
1956    35.00    12.50        12/31/1956    1956    27.18    0.127807955    273.85    27.6        1917-01-09    0.13
1957    35.00    12.50        12/31/1957    1957    28.15    0.131512533    266.13    28.4        1917-01-10    0.14
1958    35.00    12.50        12/31/1958    1958    28.92    0.133827895    261.53    28.9        1917-01-11    0.14
1959    35.00    12.50        12/31/1959    1959    29.16    0.136143256    257.08    29.4        1917-01-12    0.14
1960    35.00    12.50        12/31/1960    1960    29.62    0.137995545    253.63    29.8        1918-01-01    0.14
1961    35.00    12.50        12/31/1961    1961    29.92    0.13892169    251.94    30        1918-01-02    0.14
1962    35.00    12.50        12/31/1962    1962    30.26    0.140773979    248.63    30.4        1918-01-03    0.14
1963    35.00    12.50        12/31/1963    1963    30.62    0.143089341    244.60    30.9        1918-01-04    0.14
1964    35.00    12.50        12/31/1964    1964    31.03    0.144478557    242.25    31.2        1918-01-05    0.15
1965    35.00    12.50        12/31/1965    1965    31.56    0.147256991    237.68    31.8        1918-01-06    0.15
1966    35.00    12.50        12/31/1966    1966    32.46    0.152350787    229.73    32.9        1918-01-07    0.15
1967    35.00    12.65        12/31/1967    1967    33.4    0.15698151    222.96    33.9        1918-01-08    0.15
1968    38.94    16.23        12/31/1968    1968    34.8    0.164390666    236.87    35.5        1918-01-09    0.16
1969    40.76    16.98        12/31/1969    1969    36.67    0.174578257    233.48    37.7        1918-01-10    0.16
1970    36.07    15.03        12/31/1970    1970    38.84    0.184302775    195.71    39.8        1918-01-11    0.16
1971    41.17    16.91        12/31/1971    1971    40.51    0.190322715    216.32    41.1        1918-01-12    0.17
1972    59.00    23.58        12/31/1972    1972    41.85    0.196805727    299.79    42.5        1919-01-01    0.17
1973    97.84    39.90        12/31/1973    1973    44.45    0.213939402    457.33    46.2        1919-01-02    0.16
1974    158.96    67.96        12/31/1974    1974    49.33    0.240334523    661.41    51.9        1919-01-03    0.16
1975    160.91    72.42        12/31/1975    1975    53.84    0.257005126    626.10    55.5        1919-01-04    0.17
1976    124.71    69.05        12/31/1976    1976    56.94    0.269508078    462.73    58.2        1919-01-05    0.17
1977    147.78    84.66        12/31/1977    1977    60.61    0.287567898    513.90    62.1        1919-01-06    0.17
1978    193.39    100.75        12/31/1978    1978    65.22    0.313499947    616.87    67.7        1919-01-07    0.17
1979    304.83    143.68        12/31/1979    1979    72.57    0.355176454    858.25    76.7        1919-01-08    0.18
1980    614.61    264.20        12/31/1980    1980    82.38    0.399631394    1537.94    86.3        1919-01-09    0.18
1981    459.26    226.47        12/31/1981    1981    90.93    0.435287962    1055.07    94        1919-01-10    0.18
1982    375.28    214.38        12/31/1982    1982    96.5    0.451958564    830.34    97.6        1919-01-11    0.19
1983    423.61    279.24        12/31/1983    1983    99.6    0.469092239    903.04    101.3        1919-01-12    0.19
1984    360.50    269.77        12/31/1984    1984    103.9    0.487615131    739.31    105.3        1920-01-01    0.19
1985    317.18    244.68        12/31/1985    1985    107.6    0.506138023    626.67    109.3        1920-01-02    0.20
1986    367.72    250.66        12/31/1986    1986    109.6    0.511694891    718.63    110.5        1920-01-03    0.20
1987    446.28    272.30        12/31/1987    1987    113.6    0.534385434    835.13    115.4        1920-01-04    0.20
1988    436.79    245.20        12/31/1988    1988    118.3    0.558002121    782.77    120.5        1920-01-05    0.21
1989    380.74    232.20        12/31/1989    1989    124    0.58393417    652.03    126.1        1920-01-06    0.21
1990    383.32    214.78        12/31/1990    1990    130.7    0.619590737    618.67    133.8        1920-01-07    0.21
1991    362.10    204.65        12/31/1991    1991    136.2    0.638576701    567.04    137.9        1920-01-08    0.20
1992    343.86    194.76        12/31/1992    1992    140.3    0.657099593    523.30    141.9        1920-01-09    0.20
1993    360.00    239.68        12/31/1993    1993    144.5    0.675159413    533.21    145.8        1920-01-10    0.20
1994    384.12    250.79        12/31/1994    1994    148.2    0.693219232    554.11    149.7        1920-01-11    0.20
1995    384.05    243.31        12/31/1995    1995    152.4    0.71081598    540.29    153.5        1920-01-12    0.19
1996    387.82    248.33        12/31/1996    1996    156.9    0.734432667    528.05    158.6        1921-01-01    0.19
1997    330.98    202.10        12/31/1997    1997    160.5    0.746935619    443.12    161.3        1921-01-02    0.18
1998    294.12    177.56        12/31/1998    1998    163    0.758975499    387.52    163.9        1921-01-03    0.18
1999    278.55    172.13        12/31/1999    1999    166.6    0.77935068    357.41    168.3        1921-01-04    0.18
2000    279.10    184.09        12/31/2000    2000    172.2    0.805745801    346.39    174        1921-01-05    0.18
2001    272.67    189.36        12/31/2001    2001    177.1    0.818248753    333.24    176.7        1921-01-06    0.18
2002    309.66    206.27        12/31/2002    2002    179.9    0.83769779    369.66    180.9        1921-01-07    0.18
2003    362.91    222.20        12/31/2003    2003    184    0.853442248    425.23    184.3        1921-01-08    0.18
2004    409.17    223.36        12/31/2004    2004    188.9    0.881226586    464.32    190.3        1921-01-09    0.18
2005    444.47    244.86        12/31/2005    2005    195.3    0.911326285    487.72    196.8        1921-01-10    0.18
2006    603.95    327.68        12/31/2006    2006    201.6    0.9344799    646.30    201.8        1921-01-11    0.17
2007    695.39    347.00        12/31/2007    2007    207.34    0.972618535    714.96    210.036        1921-01-12    0.17
2008    871.65    473.17        12/31/2008    2008    215.3    0.973507634    895.37    210.228        1922-01-01    0.17
2009    972.90    621.59        12/31/2009    2009    214.54    1    972.90    215.949        1922-01-02    0.17

The models were fitted to the data using the free Octave numerical programming environment. Octave is Matlab compatible and available under the GNU Public License. Octave is available in binary as well as source code versions for Windows, Mac OS, and Linux. The standard polyfit polynomial fitting function in Octave was used to fit the polynomial models to the annual gold price data. The leasqr least squares fitting function from the optim Octave add-on package was used to fit the Gaussian and Cauchy-Lorentz peak models to the annual gold price data. Octave Forge add-on packages are available as Unix style blatz.tar.gz files. There is no need to extract the contents of these files. Octave has a command pkg install which handles installing the packages; just type pkg install blatz.tar.gz. The GNU Octave installation includes a C/C++ compiler to compile any C or C++ files included in the package. Note that the run-time errors reported by Octave, for example due to a syntax error, can be cryptic. Be patient and don’t always trust the error message.

% plot_gold.m
% Description: Octave (Matlab compatible) script to plot annual gold price data
% and fit polynomial models to the data.  Tested on Windows XP Service Pack 2
% with Octave 3.2.4 for Windows installed.
%
% Author: John F. McGowan, Ph.D.
% Copyright (C) 2010 by John F. McGowan
%
%
disp('reading gold price data...');
fflush(stdout);
data = dlmread('annual_gold_price_from_1900.txt');
disp('making plot 1');
fflush(stdout);
figure(1)
plot(data(:,1), data(:,8));
title('Inflation Adjusted Gold Price');
xlabel('Year');
ylabel('Gold Price USD (2009)');
disp('making plot 2');
fflush(stdout);
figure(2)
plot(data(:,1), data(:,2));
title('Nominal Gold Price');
xlabel('Year');
ylabel('Gold Price USD');
disp('making cpi plot');
figure(3)
plot(data(:,1), data(:,6));
title('US Consumer Price Index (CPI)');
xlabel('Year');
% annual inflation rate
disp('plotting annual inflation rate');
fflush(stdout);
figure(4)
cpi = data(:,6);
cpi_shift = shift(cpi,1);
inflation = (cpi - cpi_shift) ./ cpi_shift;
years = data(:,1);
plot(years(2:end), inflation(2:end)*100);
title('US Annual Inflation Rate');
xlabel('Year');
ylabel('Inflation (%)');
%
% fitting polynomial model to data 1970 to 2009
%
disp('making gold price prediction 12 terms');
fflush(stdout);
figure(5)
floating = years(71:end);
floating_gold_price = data(71:end, 8);
p = polyfit(floating, floating_gold_price, 12);
prediction = (1970:2020);
f = polyval(p, prediction);
plot(floating, floating_gold_price, 'o', prediction, f, '-');
title('Polynomial Model of Gold Price (12 Terms)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
disp('making gold price prediction 24 terms');
fflush(stdout);
figure(6)
floating = years(71:end);
floating_gold_price = data(71:end, 8);
p = polyfit(floating, floating_gold_price, 24);
prediction = (1970:2020);
f = polyval(p, prediction);
plot(floating, floating_gold_price, 'o', prediction, f, '-');
title('Polynomial Model of Gold Price (24 Terms)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
disp('making gold price prediction 32 terms');
fflush(stdout);
figure(7)
floating = years(71:end);
floating_gold_price = data(71:end, 8);
p = polyfit(floating, floating_gold_price, 32);
prediction = (1970:2020);
f = polyval(p, prediction);
plot(floating, floating_gold_price, 'o', prediction, f, '-');
title('Polynomial Model of Gold Price (32 Terms)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
disp('all done');
fflush(stdout);

% THE END

The script to fit the two Gaussian peak model to the data is:

% fit_gold.m
% Description: Octave (matlab compatible) script to fit two gaussian peak model
% to annual gold price data using Octave 3.2.4 and the optim package version 1.0.15
% Tested on Windows XP Service Pack 2 with Octave 3.2.4 and optim 1.0.15 installed.
%
% Author: John F. McGowan, Ph.D.
% Copyright (C) 2010 by John F. McGowan
%
disp('fitting model to gold price data');
fflush(stdout);
% fit model to gold price data
%floating_gold_price has inflation adjusted gold price since 1970
floating_years = years(71:end);

% Define functions
% model annual gold price data as two Gaussians
leasqrfunc = @(x,p) p(1) * exp(-1.0*(x - p(2)).^2/p(3)^2) + p(4) * exp(-1.0*(x - p(5)).^2/p(6)^2);
leasqrdfdp = @(x, f, p, dp, func) [exp(-1.0*(x - p(2)).^2/p(3)^2), (2*(x - p(2))/p(3)^2) * p(1) .* exp(-1.0*(x - p(2)).^2/p(3)^2), (2 * (x - p(2)).^2/p(3)^3) * p(1) .* exp(-1.0*(x - p(2)).^2/p(3)^2), exp(-1.0*(x - p(5)).^2/p(6)^2), (2*(x - p(5))/p(6).^2) * p(4) .* exp(-1.0*(x - p(5)).^2/p(6)^2), (2 * (x - p(5)).^2/p(6)^3) * p(4) .* exp(-1.0*(x - p(5)).^2/p(6)^2) ];

wt1 = ones(size(floating_gold_price));
t = floating_years;
data = floating_gold_price;

F = leasqrfunc;
dFdp = leasqrdfdp; % exact derivative
dp = [50.0; 1.0; 1.0; 50.0; 1.0; 1.0];
pin = [500.0; 1980.; 5.0; 500.0; 2010.0; 5.0 ];
stol=0.01; niter=50;
minstep = [10.0; 0.2; 0.2; 10.0; 0.2; 0.2];
maxstep = [100.0; 5.0; 5.0; 100.0; 5.0; 5.0];
options = [minstep, maxstep];

disp(size(t));
disp(size(data));
disp(size(wt1));
fflush(stdout);
figure(1);
global verbose;
verbose = 1;
[f1, p1, kvg1, iter1, corp1, covp1, covr1, stdresid1, Z1, r21] = ...
leasqr (t, data, pin, F, stol, niter, wt1, dp, dFdp, options);

%
% make a prediction
figure(2);
pred_years = [1970:2020];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
figure(3);
pred_years = [1970:2050];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
figure(4);
pred_years = [1970:2100];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');

data_2G = data;
model_2G = leasqrfunc(floating_years, p1);
diff_2G = model_2G - data_2G;
chisq_2G = diff_2G' * diff_2G;

disp('all done');
% THE END

The script to fit the seven (7) Gaussian peak model to the annual gold price data is:

% fit_gold7.m
% Description: Octave (matlab compatible) script to fit seven (7) Gaussian peak model
% to annual gold price data.
% Tested using Octave 3.2.4 with optim 1.0.15 add on package installed on Windows XP Service Pack 2
% Author: John F. McGowan, Ph.D.
% Copyright (C) John F. McGowan
%
disp('fitting 7 Gaussian model to gold price data');
fflush(stdout);
% fit model to gold price data
%floating_gold_price has inflation adjusted gold price since 1970
floating_years = years(71:end);

% Define functions

% model as seven (7) un-normalized Gaussians
leasqrfunc = @(x,p) p(1) * exp(-1.0*(x - p(2)).^2/p(3)^2) + p(4) * exp(-1.0*(x - p(5)).^2/p(6)^2) + p(7) * exp(-1.0*(x - p(8)).^2/p(9)^2) + p(10) * exp(-1.0*(x - p(11)).^2/p(12)^2) + p(13) * exp(-1.0*(x - p(14)).^2/p(15)^2) + p(16) * exp(-1.0*(x - p(17)).^2/p(18)^2) + p(19) * exp(-1.0*(x - p(20)).^2/p(21)^2);

leasqrdfdp = @(x, f, p, dp, func) [exp(-1.0*(x - p(2)).^2/p(3)^2), (2*(x - p(2))/p(3)^2) * p(1) .* exp(-1.0*(x - p(2)).^2/p(3)^2), (2 * (x - p(2)).^2/p(3)^3) * p(1) .* exp(-1.0*(x - p(2)).^2/p(3)^2), exp(-1.0*(x - p(5)).^2/p(6)^2), (2*(x - p(5))/p(6).^2) * p(4) .* exp(-1.0*(x - p(5)).^2/p(6)^2), (2 * (x - p(5)).^2/p(6)^3) * p(4) .* exp(-1.0*(x - p(5)).^2/p(6)^2), exp(-1.0*(x - p(8)).^2/p(9)^2), (2*(x - p(8))/p(9)^2) * p(7) .* exp(-1.0*(x - p(8)).^2/p(9)^2), (2 * (x - p(8)).^2/p(9)^3) * p(7) .* exp(-1.0*(x - p(8)).^2/p(9)^2) , exp(-1.0*(x - p(11)).^2/p(12)^2), (2*(x - p(11))/p(12)^2) * p(10) .* exp(-1.0*(x - p(11)).^2/p(12)^2), (2 * (x - p(11)).^2/p(12)^3) * p(10) .* exp(-1.0*(x - p(11)).^2/p(12)^2) , exp(-1.0*(x - p(14)).^2/p(15)^2), (2*(x - p(14))/p(15)^2) * p(13) .* exp(-1.0*(x - p(14)).^2/p(15)^2), (2 * (x - p(14)).^2/p(15)^3) * p(13) .* exp(-1.0*(x - p(14)).^2/p(15)^2), exp(-1.0*(x - p(17)).^2/p(18)^2), (2*(x - p(17))/p(18)^2) * p(16) .* exp(-1.0*(x - p(17)).^2/p(18)^2), (2 * (x - p(17)).^2/p(18)^3) * p(16) .* exp(-1.0*(x - p(17)).^2/p(18)^2) , exp(-1.0*(x - p(20)).^2/p(21)^2), (2*(x - p(20))/p(21)^2) * p(19) .* exp(-1.0*(x - p(20)).^2/p(21)^2), (2 * (x - p(20)).^2/p(21)^3) * p(19) .* exp(-1.0*(x - p(20)).^2/p(21)^2) ];

wt1 = ones(size(floating_gold_price));
t = floating_years;
data = floating_gold_price;

F = leasqrfunc;
dFdp = leasqrdfdp; % exact derivative
dp = [50.0; 1.0; 1.0; 50.0; 1.0; 1.0; 50.0; 1.0; 1.0; 50.0; 1.0; 1.0 ; 50.0; 1.0; 1.0; 50.0; 1.0; 1.0; 50.0; 1.0; 1.0];
pin = [500.0; 1980.; 5.0; 500.0; 2010.0; 5.0; 500.0; 1982; 2.0; 500.0; 1987; 2.0; 500.0; 1995; 5.0 ; 500.0; 1974; 5.0; 500.0; 1974; 5.0 ];
stol=0.01; niter=100;
minstep = [10.0; 0.2; 0.2; 10.0; 0.2; 0.2; 10.0; 0.2; 0.2; 10.0; 0.1; 0.1; 10.0; 0.1; 0.1; 10.0; 0.1; 0.1; 10.0; 0.1; 0.1];
maxstep = [100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0];
options = [minstep, maxstep];

disp(size(t));
disp(size(data));
disp(size(wt1));
fflush(stdout);
figure(1);
global verbose;
verbose = 1;
[f1, p1, kvg1, iter1, corp1, covp1, covr1, stdresid1, Z1, r21] = ...
leasqr (t, data, pin, F, stol, niter, wt1, dp, dFdp, options);

%
% make a prediction
figure(2);
pred_years = [1970:2020];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (7 Gaussian Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
figure(3);
pred_years = [1970:2050];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (7 Guassian Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
figure(4);
pred_years = [1970:2100];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (7 Gaussian Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');

data_7G = data;
model_7G = leasqrfunc(floating_years, p1);
diff_7G = model_7G - data_7G;
chisq_7G = diff_7G' * diff_7G;

disp('all done (7 Gaussian Fit) all done');
% THE END

The script to fit the two Cauchy-Lorentz function peak model to the annual gold price data is:

% fit_gold_cauchy.m
% Description: Octave (matlab compatible) script to fit a two Cauchy-Lorentz peak model to the annual gold price data.  Tested using Octave 3.2.4 and the optim 1.015 add-on package on Windows XP Service Pack 2.
% Author: John F. McGowan, Ph.D.
% Copyright (C) John F. McGowan
%
disp('fitting 2 Cauchy-Lorentz model to gold price data');
fflush(stdout);
% fit model to gold price data
%floating_gold_price has inflation adjusted gold price since 1970
floating_years = years(71:end);

% Define functions

% model as the linear combination of two Cauchy-Lorentz (aka Breit-Wigner) functions
leasqrfunc = @(x,p) p(1) ./(1 + (x - p(2)).^2/p(3)^2) + p(4) ./(1 + (x - p(5)).^2/p(6)^2);
leasqrdfdp = @(x, f, p, dp, func) [1.0 ./(1.0 + (x - p(2)).^2/p(3)^2), (2*p(1)*(x-p(2)))./(p(3)^2*((x-p(2)).^2/p(3)^2+1).^2), (2*p(1)*(x-p(2)).^2)./(p(3)^3*((x-p(2)).^2/p(3)^2+1).^2), 1.0 ./(1.0 + (x - p(5)).^2/p(6)^2), (2*p(4)*(x-p(5)))./(p(6)^2*((x-p(5)).^2/p(6)^2+1).^2), (2*p(4)*(x-p(5)).^2)./(p(6)^3*((x-p(5)).^2/p(6)^2+1).^2)];

wt1 = ones(size(floating_gold_price));
t = floating_years;
data = floating_gold_price;

F = leasqrfunc;
dFdp = leasqrdfdp; % exact derivative
dp = [50.0; 1.0; 1.0; 50.0; 1.0; 1.0];
pin = [500.0; 1980.; 5.0; 500.0; 2010.0; 5.0 ];
stol=0.01; niter=50;
minstep = [10.0; 0.2; 0.2; 10.0; 0.2; 0.2];
maxstep = [100.0; 5.0; 5.0; 100.0; 5.0; 5.0];
options = [minstep, maxstep];

disp(size(t));
disp(size(data));
disp(size(wt1));
fflush(stdout);
figure(1);
global verbose;
verbose = 1;
[f1, p1, kvg1, iter1, corp1, covp1, covr1, stdresid1, Z1, r21] = ...
leasqr (t, data, pin, F, stol, niter, wt1, dp, dFdp, options);

%
% make a prediction
figure(2);
pred_years = [1970:2020];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (2 Cauchy Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
figure(3);
pred_years = [1970:2050];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (2 Cauchy Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
figure(4);
pred_years = [1970:2100];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (2 Cauchy Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');

% C suffix for Cauchy Lorentz model
data_2C = data;
model_2C = leasqrfunc(floating_years, p1);
diff_2C = model_2C - data_2C;
chisq_2C = diff_2C' * diff_2C;

disp('all done');
% THE END

The Octave script to fit the seven (7) Cauchy-Lorentz peak model to the annual gold price data is:

% fit_gold7.m
% Description: Octave (matlab compatible) script to fit seven (7) Cauchy-Lorentz peak model
% to annual gold price data.
% Tested using Octave 3.2.4 with optim 1.0.15 add on package installed on Windows XP Service Pack 2
% Author: John F. McGowan, Ph.D.
% Copyright (C) John F. McGowan
%
disp('fitting seven (7) Cauchy-Lorentz model to gold price data');
fflush(stdout);
% fit model to gold price data
%floating_gold_price has inflation adjusted gold price since 1970
floating_years = years(71:end);

% Define gold price model functions

% model as linear combination of seven (7) Cauchy-Lorentz (aka Breit-Wigner) functions
leasqrfunc = @(x,p) p(1) ./(1 + (x - p(2)).^2/p(3)^2) + p(4) ./(1 + (x - p(5)).^2/p(6)^2) + p(7) ./(1 + (x - p(8)).^2/p(9)^2) + p(10) ./(1 + (x - p(11)).^2/p(12)^2) + p(13) ./(1 + (x - p(14)).^2/p(15)^2) + p(16) ./(1 + (x - p(17)).^2/p(18)^2) + p(19) ./(1 + (x - p(20)).^2/p(21)^2);

leasqrdfdp = @(x, f, p, dp, func) [1.0 ./(1.0 + (x - p(2)).^2/p(3)^2), (2*p(1)*(x-p(2)))./(p(3)^2*((x-p(2)).^2/p(3)^2+1).^2), (2*p(1)*(x-p(2)).^2)./(p(3)^3*((x-p(2)).^2/p(3)^2+1).^2),1.0 ./(1.0 + (x - p(5)).^2/p(6)^2), (2*p(4)*(x-p(5)))./(p(6)^2*((x-p(5)).^2/p(6)^2+1).^2), (2*p(4)*(x-p(5)).^2)./(p(6)^3*((x-p(5)).^2/p(6)^2+1).^2),1.0 ./(1.0 + (x - p(8)).^2/p(9)^2), (2*p(7)*(x-p(8)))./(p(9)^2*((x-p(8)).^2/p(9)^2+1).^2), (2*p(7)*(x-p(8)).^2)./(p(9)^3*((x-p(8)).^2/p(9)^2+1).^2),1.0 ./(1.0 + (x - p(11)).^2/p(12)^2), (2*p(10)*(x-p(11)))./(p(12)^2*((x-p(11)).^2/p(12)^2+1).^2), (2*p(10)*(x-p(11)).^2)./(p(12)^3*((x-p(11)).^2/p(12)^2+1).^2),1.0 ./(1.0 + (x - p(14)).^2/p(15)^2), (2*p(13)*(x-p(14)))./(p(15)^2*((x-p(14)).^2/p(15)^2+1).^2), (2*p(13)*(x-p(14)).^2)./(p(15)^3*((x-p(14)).^2/p(15)^2+1).^2),1.0 ./(1.0 + (x - p(17)).^2/p(18)^2), (2*p(16)*(x-p(17)))./(p(18)^2*((x-p(17)).^2/p(18)^2+1).^2), (2*p(16)*(x-p(17)).^2)./(p(18)^3*((x-p(17)).^2/p(18)^2+1).^2),1.0 ./(1.0 + (x - p(20)).^2/p(21)^2), (2*p(19)*(x-p(20)))./(p(21)^2*((x-p(20)).^2/p(21)^2+1).^2), (2*p(19)*(x-p(20)).^2)./(p(21)^3*((x-p(20)).^2/p(21)^2+1).^2)]

wt1 = ones(size(floating_gold_price));  % same weight for all data points
t = floating_years;    % years when gold price floats
data = floating_gold_price;   % inflation adjusted gold price

F = leasqrfunc;
dFdp = leasqrdfdp; % exact derivative
dp = [50.0; 1.0; 1.0; 50.0; 1.0; 1.0; 50.0; 1.0; 1.0; 50.0; 1.0; 1.0 ; 50.0; 1.0; 1.0; 50.0; 1.0; 1.0; 50.0; 1.0; 1.0];
pin = [500.0; 1980.; 5.0; 500.0; 2010.0; 5.0; 500.0; 1982; 2.0; 500.0; 1987; 2.0; 500.0; 1995; 5.0 ; 500.0; 1974; 5.0; 500.0; 1974; 5.0 ];
stol=0.01; niter=100;
minstep = [10.0; 0.2; 0.2; 10.0; 0.2; 0.2; 10.0; 0.2; 0.2; 10.0; 0.1; 0.1; 10.0; 0.1; 0.1; 10.0; 0.1; 0.1; 10.0; 0.1; 0.1];
maxstep = [100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0; 100.0; 5.0; 5.0];
options = [minstep, maxstep];

disp(size(t));
disp(size(data));
disp(size(wt1));
fflush(stdout);
figure(1);
global verbose;
verbose = 1;
[f1, p1, kvg1, iter1, corp1, covp1, covr1, stdresid1, Z1, r21] = ...
leasqr (t, data, pin, F, stol, niter, wt1, dp, dFdp, options);

%
% make a prediction
figure(2);
pred_years = [1970:2020];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (7 Cauchy Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
figure(3);
pred_years = [1970:2050];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (7 Cauchy Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');
%
disp('plotting figure 4');
fflush(stdout);
figure(4);
pred_years = [1970:2100];
prediction = leasqrfunc(pred_years, p1);
plot(floating_years, data, 'o', pred_years, prediction, '-');
title('Inflation Adjusted Gold Price (7 Cauchy Prediction vs Data)');
xlabel('Year');
ylabel('Gold Price USD (2009)');

disp('computing final results...');
fflush(stdout);
% C suffix for Cauchy Lorentz model
data_7C = data;
model_7C = leasqrfunc(floating_years, p1);
diff_7C = model_7C - data_7C;
chisq_7C = diff_7C' * diff_7C;

disp('all done');
% THE END

The derivatives of the models with respect to parameters were computed with the symbolic manipulation package Maxima. Maxima was also used to generate some of the Latex mathematical formulas in this article.

Many of the legends have become part of history, others are history in the making. Pappus (“old man”) of Alexandria is the nameless giant and last of the ancient geometricians. Charles Babbage was the last of the Industrial Revolution mathematics machine-makers. Who, after watching “A beautiful mind” will ever forget the story of John Forbes Nash, Jr. battling his paranoid schizophrenia, not with pills but with logic and his winning a Nobel prize in Economic Science? And then there are the truly monumental life sagas of the largely unknown: Srinivasa Ramanujan, Jakow Trachtenberg, Andrew Wiles and Grigori Perelman.

A young, self-taught Ramanujan, set sail for England in a merchant ship in 1914 just before the outbreak of WWI and became the first Indian to be elected a Fellow of Trinity College, Cambridge just before the war’s end in 1918. Trachtenberg, imprisoned for 7 years in Nazi concentration camps, without pen or paper, worked mentally and developed speed arithmetic. The mathematics professor Wiles kept a 7 year-secret, working in recluse day and night and finally proved the 17th Century Fermat’s Last Theorem in 1994. Perhaps most mysterious of them all is Perelman, the Russian genius who posted a series of eprints to the free arXiv server in 2002, that proved the 1904 Poincaré Conjecture. He continues to live in poverty refusing the $1,000,000 Clay Millenium Prize awarded to him. Each of their stories is so impressive that each one should be made into a movie. Then there are the legends we ourselves create. We all have our personal heroes and heroines. Each of them a true legend.

And then there are the math myths. We are surrounded by many inspiring minds but the myths seem to linger on. Many have tried to dispel them. Perhaps no one more that Paul Halmos, a Hungarian-born American mathematician. At the University of Connecticut’s Gallery of Mathematicians, his portrait shares a wall with other great wizards such as Archimedes, Descartes, Euclid, Galileo and Newton. Not bad, at all. But I can hear you saying, “who the hell was he”? Heard of Q.E.D. (“quod erat demonstrandum”)? It’s what mathematicians write at the end of proof. The more common end-of-proof mark is “∎” which is Unicode symbol U+220E also known as “the Halmos”. He made fundamental advances in the areas of probability theory, statistics, Hilbert spaces and algebraic structure of mathematics and won medals for his ability to communicate mathematics. His “I Want to Be a Mathematician: An Automathography” is well worth a read.

Five of the most common math myths are:

1. The Genius Myth (that good mathematicians are born with special math talent and enormous left brains);
2. The Good Memory Myth (that good mathematicians have a phenomenal memory for formulas);
3. The Using-Tools-Is-Cheating Myth (that good mathematicians don’t use fingers, toes and calculators);
4. The Gender Myth (that good mathematicians are all men despite the abundance of female bio-statisticians);
5. The Who Needs it Anyway Myth (that math is useful only to mathematicians).

But the biggest myth of them all is the “I-Cant-Do-Math Myth”. I recently taught multivariate calculus to a class of non-mathematicians and social scientists. It wasn’t just them asking the question “why are we here”? But an open-mind is a powerful adversary. They soon dusted-off this myth in a matter of months. Yes, for some, math is like sorcery. We all have our superstitions to overcome. The good news is that we can. Arthur C Clarke who once said that, “any sufficiently advanced technology is indistinguishable from magic”. If that technology is born of math then however miraculous or foreboding it appears, we will learn to embrace it. Mathematics – to learn. Let’s face our fears, dispel the myths and advance. There are legends to be made.

Dr Michael Taylor (PhD, CPhys)
National Observatory of Athens &
American University of Athens
http://patternizer.wordpress.com/

Sponsor’s message: Receive free weekly updates about new math books. Don’t miss great new titles in the genres you love (such as Mathematics, Science, Programming, and Sci-Fi): http://anynewbooks.com