Jonathan Borwein and Keith Devlin are well-known mathematicians who have a strong appreciation of, and expertise in, experimental mathematics. In this book they provide us with a concise, inviting introduction to the field.

The first chapter tries to succinctly explain what experimental mathematics is and why it’s a fundamental tool for the modern mathematician. The following is their definition:

Experimental mathematics is the use of a computer to run computations—sometimes no more than trial-and-error tests—to look for patterns, to identify particular numbers and sequences, to gather evidence in support of specific mathematical assertions that may themselves arise by computational means, including search. Like contemporary chemists—and before them the alchemists of old—who mix various substances together in a crucible and heat them to a high temperature to see what happens, today’s experimental mathematician puts a hopefully potent mix of numbers, formulas, and algorithms into a computer in the hope that something of interest emerges.

They immediately address some of the possible objections and illustrate how an approach that doesn’t focus on formal proof, but rather on exploration and experimentation, ultimately leads to hypotheses which can then be, in many cases, proved analytically. The authors argue that in this sense, thanks to the aid of advanced computers, mathematics is becoming more and more similar to other natural sciences.

They also make a case for how great mathematicians like Euler, Gauss, and Reimann were doing experimental mathematics well before calculators where available. Their calculations on paper were far more limited than what computers afford us these days, yet they served them well when it came to sharpening and verifying their intuitions.

The rest of the book is a continuous series of examples that show the advantages of this approach in practice. The examples are highly interesting (some of them stunning) and tend to focus on calculus, analysis and analytical number theory.

Each chapter is accompanied by a section called “Explorations”. I found this section to be particularly valuable. Within it you’ll find exercises, and further examples and considerations. The answers/solutions to the actual problems are provided in the second to last chapter, just before the brief epilogue.

Chapter 2 discusses how to calculate an arbitrary digit for irrational numbers like , in certain bases. They illustrate how the so called BBP Formula (Bailey-Borwein-Plouffe formula, co-discovered by Jonathan Borwein’s brother) came to be.

The use of a program which implements the PSQL integer relation algorithm in high-precision, floating-point arithmetic was key to its discovery. The BBP Formula in turn allowed the calculation of the quadrillionth binary digit of back in 2000.

Chapter 3 focuses on identifying numbers, digits patterns, and sequences once you obtain a numeric result through your calculations and experimentation. They introduce the subject with relatively obvious values like the approximations of or , but the chapter quickly escalates to an example where a closed form for a seemingly random sequence needs to be found.

Chapter 4 analyzes the Reimann Zeta function from the eyes of an experimental mathematician, and shows us what kind of insight we can gain from this unique perspective.

In chapter 5 we learn how by numerically evaluating definite integrals, it is sometimes possible to identify the resulting value which will help us to analytically resolve those particular integrals. The examples presented in this chapter originate for the most part from physics and are very challenging if attempted without the aid of experimental methods. To better grasp the kind of integrals discussed in this chapter, here is an example:

The explorations section provides a few more interesting integrals, including some for which a closed form is not known. The authors even include an integral that intentionally stumps Mathematica 6 and Maple 11.

Chapter 6 is dedicated to serendipitous discoveries (“proof by serendipity”) with a few interesting examples of how “luck” met preparation, ultimately enriching the body of mathematical knowledge almost by chance.

In chapter 7 the authors go back to talk about , this time in base 10, to calculate its digits with efficient, fast converging formulas and methods. The chapter wraps up with a discussion about the normality of , which hasn’t been proved of course, but appears to be empirically supported by the statistical analysis of the first trillion digits. In the explorations section there is a nice discussion about the implementation of fast arithmetic through the Karatsuba multiplication, and the subject of Montecarlo simulations (a very inefficient method of calculating , but a great way to show the idea behind Montecarlo simulations).

Chapter 8 has a bold title, “The computer knows more math than you do”. This provocative title is quickly diminished to put it in context though. The authors start by approaching a tough problem posed by Donald Knuth (of TeX and The Art of Computer Programming fame) to the readers of the American Mathematical Monthly:

In an attempt to solve this the authors invite us to go on a journey involving the Lambert W function, the Pochhammer function, and Abel’s limit theorem. The rest of the chapter illustrates another difficult problem whose solution obtained through the aid of Maple has important implications not only for mathematics, but also for quantum field theory and statistical mechanics.

In chapter 9 a few infinite series are calculated in order to show how CAS systems and experimental methodology can still be useful when dealing with problems that involve infinite sequences, series, and products.

Chapter 10 is dedicated to the limits and the dangers of this approach. Several examples showcase how one can be misled into making assumptions, and how to avoid this from happening. The ad hoc example below is correct to over half a billion digits:

After having calculated a few hundred digits, it would be natural to assume that the series converges to a natural number, when in reality it’s an irrational and transcendental number.

In chapter 11, conscious of the selective focus on analysis and analytical number theory throughout the book, Borwein and Devlin introduce other examples such as a topology problem whose proof was reached thanks to a deeper insight gained through computer visualization of a surface, a knot theory problem, the Four Color Theorem, the Robbins Conjecture, the computation of , and so on.

In truth, I feel that such a thin book could have used more examples like the ones in chapter 11, in order to make a stronger case for the applicability of experimental mathematics to areas outside of analysis.

The book is well written and the tone is never heavy, despite the advanced mathematical examples within it. The authors include historical background and anecdotes which makes for a more interesting read and provides a human perspective behind the formulas presented. The (at times) funny illustrations and occasional jokes are definitely a pleasant addition.

This book is relatively tool agnostic; Maple and Mathematica are referenced throughout, and so are a few online tools to identify number sequences and known numeric values. Overall though, the emphasis in on the methodology rather than a particular CAS (Computer Algebra System) or programming language. In fact, with the exception of a snippet of Maple code in one of the explorations in the first chapter, the book describe the examples from a mathematical and algorithmic standpoint. You won’t find source code for the examples illustrated.

The ideal target audience for The Computer as Crucible is graduate students and researchers. A bright, motivated high-school student will get the gist of this book, but a more mature mathematical audience will actually be able to follow the steps within the examples and fully appreciate the insight on how an experimental approach can aid their research.

Despite the numerous examples employed to make their case, the authors start the book by explaining that it is not intended to be comprehensive. It’s meant to be thought provoking and to whet your appetite as to what is now possible in mathematical research thanks to computers.

As a computer programmer who’s passionate about mathematics, experimental mathematics fascinates me greatly. As such, I hope to work my way through the actual textbooks that are generally suggested as a follow up to this book. Namely, I’ve already started reading Mathematics by Experiment: Plausible Reasoning in the 21st Century (Second Edition), which is co-authored by Jonathan Borwein himself. Other textbooks referenced in this introduction are Experimental Mathematics in Action and Experimentation in Mathematics: Computational Paths to Discovery.

In conclusion, The Computer as Crucible is a lovely little book which builds a strong case for experimental mathematics. Any practicing mathematician or serious amateur should consider checking out this introduction to a topic that will no doubt transform mathematics.

Full disclosure: We received this book for free from the publisher, but we’re under no obligation to review or endorse it. We routinely receive a fair number of books from several publishers that never make the cut for an actual review. The links have our Amazon referral id which gives us a tiny percentage if you buy a book. In turn this helps support this site.

In Kepler’s time the reigning Ptolemaic theory could predict the position of Mars to within a few degrees, usually less than a one percent error. How important is such a small error? Space missions routinely depend on modern orbital dynamics, a lineal descendant of Kepler’s work, to make far more accurate calculations to succeed. The Mars Climate Orbiter mission in 1999 failed due to a tiny error. After traveling about 300 million miles, the Mars Climate Orbiter came in about 90 miles, a tiny fraction of 300 million miles, too low, burning up in the Martian atmosphere rather than aerobreaking successfully into orbit. Successful space missions, the Global Positioning System (GPS), and other modern applications depend on precision mathematical models similar to and sometimes directly descended from Kepler’s model of the orbit of Mars.

Kepler’s story is very different from the story of Galileo and it offers different lessons for today. Diverse fields ranging from astronomy and space physics to artificial intelligence are confronted with similarly complex and confusing data. A mathematical solution to an outstanding problem comparable to Kepler’s discovery could reveal long suspected connections between gravity and other forces, perhaps enabling new power or propulsion systems, enable computers to recognize objects and spoken words, or solve other problems. This article will discuss the discovery of the elliptical orbit of Mars in the context of Kepler’s time. It will also draw some lessons from Kepler and compare and contrast Kepler’s process of discovery to modern astronomy, physics, space science and engineering, including a detailed discussion of dark matter and dark energy.

Read the rest of Kepler’s New Astronomy on Scribd, where you’ll be able to download it in several formats including PDF, or click Fullscreen in the embedded document below.

PDF: Kepler’s New Astronomy.pdf

The dream search engine would search by topic, by the detailed content of the items searched, ideally finding the desired information immediately. Actual understanding of text remains a unfulfilled promise of artificial intelligence. Statistical language processing can achieve a degree of searching by topic. This article introduces the basic concepts and mathematics of statistical language processing and its applications to search. It gives a brief introduction and overview of more advanced techniques in statistical language processing as applied to search. It also includes sample Ruby code illustrating some simple statistical language processing methods.

Read the rest of Faster, Better, Cheaper Search Engines on Scribd, where you’ll be able to download it in several formats including PDF, or click Fullscreen in the embedded document below.

Source code: trigram.zip

PDF: Faster-Better-Cheaper-Search-Engines.pdf

The author of the site, describes the project as follows:

Project Euler is a series of challenging mathematical/computer programming problems that will require more than just mathematical insights to solve. Although mathematics will help you arrive at elegant and efficient methods, the use of a computer and programming skills will be required to solve most problems.

When you first register, you will be presented with 252 problems available to be solved. This number changes frequently, however, as new problems are routinely created and added to the list. The solutions to some problems veer towards the obvious, while others require advanced mathematics to tackle. Likewise, a few can be solved with paper and pencil and a good dose of insight. Most however, will require the aid of a computer program written in the language of your choice.

A sample problem (187)

Each program you execute should output your result within a minute. That’s part of the challenge, even though there is no way for the site to enforce this rule.

When you think you’ve found the correct result for a given problem, you can fill in the result field for the problem, and see if you got it right. If you have, your answer will be added to the list of problems you solved and will count towards your overall score, while bumping up your position on the participant scoreboards. There are several scoreboards, depending on what level you are at, which in turn is determined by the number of problems you have solved so far.

More importantly, solving a problem will gain you access to the respective forum (for that problem), where you’ll be able to see solutions and discussions from other participants, who may have used different languages and solution strategies.

Project Euler is appealing for a variety of reasons. In particular:

• It allows you to explore areas of math that you might not be familiar with. Solving certain problems may require you do a fair amount of research and in turn help you to learn more about certain branches of mathematics;
• If you are a math major who is trying to hone their programming skills, you’ll find Project Euler to be challenging, fun, and an excellent opportunity for improvement. Having to implement several small programs and being able to compare your style with those of other people in the forum, will definitely end up improving your knowledge of many algorithms and your skills as a programmer;
• The site’s one minute rule has an important consequence. The computational complexity of your programs really ends up mattering. While some simple problems can in fact be brute forced, the majority of them require you to write faster algorithms and take advantage of mathematical insight so as to improve the performance of your program. For certain problems a naive approach would take more than a lifetime on current PCs;
• Finally, if you are a programmer who’s scoping out a new programming language, be it Python, Ruby, Scala, Haskell or Erlang, you’ll find a great ally in Project Euler. Having to write hundreds of programs in a given language, will naturally increase your familiarity with that language. And again, comparing your newcomer coding style with those of more experience participants, will no doubt contribute to your advancement within the given language you’re focusing on.

Project Euler truly deserves five stars. I recommend it highly to anyone who is unafraid of writing a few lines of code in order to solve interesting math problems.

Complex algorithms are typically implemented as either computer software or in custom VLSI chips (e.g. Application Specific Integrated Circuits or ASICs). Research and development of complex algorithms is a specialized area and differs in a number of ways from most software and hardware development. Remarkably, many computer software business and engineering professionals often underestimate or do not understand the difficulties and scope of complex algorithm projects.

Complex algorithms are already in widespread use in commercial applications. Prominent examples include the video compression algorithms that enable BluRay, DVD Video, YouTube, and many other modern digital video systems. The US DVD Video market is around $25 billion per year (2007). Although limited, speech recognition such as now frequently encountered in telephone help and customer service systems is another example. Other examples include encryption, seismic modeling used in oil and gas exploration, sophisticated financial models, traffic models, and many others.

Complex algorithms may solve a range of major problems confronting the human race including major diseases such as cancer, the need for more and cheaper energy, and so forth. Molecular modeling may enable the design of drugs or systems of drugs that can selectively target and destroy cancer cells based on the identifying characteristics of cancer cells such as chromosomal anomalies, something currently impossible. Electromagnetic modeling software may enable the successful design and fabrication of working commercial fusion power sources. These are potential trillion-dollar markets. The global annual energy market is over $1 Trillion.

Note: Since I am a software developer, I will focus primarily on complex algorithms developed and implemented as computer software. Many of my comments apply equally well to hardware implementation. Where I have some knowledge and experience, I will make some comments on specific hardware issues.

Complex Algorithms Are Difficult

Complex algorithms are usually quite difficult to develop and often take longer than experience with other software projects would indicate. Although there are exceptions, complex algorithm projects usually take between four (4) months and several years. True research projects in which new mathematical or logical methods are developed are extremely unpredictable and typically take years. Most major scientific discoveries and inventions have taken at least five (5) years.

Complex algorithms frequently involve a tight coupling between different parts of the algorithm. All parts must work together within tight tolerances for the entire algorithm to work. This is similar to mechanical systems such as automobile engines or mechanical clocks. Indeed, implementations of complex algorithms are often referred to as “engines”, partly for this reason. Complex algorithms are often very unforgiving. Even very small errors, getting a single bit wrong, results in the implementation failing. This occurs frequently with encryption where usually every bit must be correct and video compression where even small errors often result in unacceptable “artifacts” in the decoded video. In practical terms, this means that the amount of time spent per line of working code is often significantly larger for complex algorithm projects than other software projects such as web sites, user interfaces, database reporting systems, and so forth.

Most commercial software projects involve at most mathematics taught in early high school (9th, 10th grade) in the US. Even advanced high school mathematics such as the solution to quadratic equations is uncommon outside of computer graphics. Complex algorithms in widespread use today typically involve mathematics that is taught in the first and second year of college at a good college or university in the US. A few complex algorithms involve more advanced mathematics. For example, the Global Positioning System (GPS) uses General Relativity, advanced undergraduate or graduate level mathematics, to determine the location and time correctly. In the future, more advanced mathematics may be needed for pattern recognition and other advanced tasks. Most commercial software developers do not have much experience with mathematical software at the level found in complex algorithms. Research and development of complex algorithms often requires a range of mathematical and logical skills that are not common.

Return on Investment

The return on investment for a successful complex algorithm project can be very high. Complex algorithm research and development is typically done by small teams or individuals. Small teams are the most common. Even a multi-year project, for example five years, with a ten person team (a large team) has a total cost of about $7.5 million (using a total cost per full time employee of $150K/year). A home run can solve a billion dollar or larger problem, bringing in hundreds of millions or even billions of dollars.

• Return = $100 M / $7.5 M = 13.3 (small home run)
• Return = $1 B / $7.5 M = 133 (big home run)
• Return = $1 T / $7.5 M = 133,000 (off the charts)

The greatest opportunities and the greatest risks lie in areas that require development of new mathematical or logical methods; that is true research. New complex algorithms can be converted very rapidly to commercial software products, even in a matter of months, as happened with new video compression algorithms in 2003.

Research and Development

The commercial software industry focuses overwhelmingly on “technically feasible” projects. Many venture capital firms explicitly claim to only invest in proven, technically feasible projects. Similar thinking pervades the commercial software industry. Where complex algorithms are concerned, technically feasible means proven algorithms for which working prototypes exist somewhere. The working prototypes are usually computer programs, often slow, that successfully implement the algorithm. These are frequently prototypes in the C or C++ programming languages, although Java is becoming more common (see the discussion of software engineering below). Thus, most commercial projects in the complex algorithms arena involve such tasks as porting algorithms to a different platform (for example, Unix to Windows), optimizing the algorithms for a new platform, integrating the algorithms into an application program such as a media player, converting a prototype into a production system, and so forth. Most research scientists would call these activities “Development” and not “Research” or “Research and Development”.

The commercial software industry follows a widely accepted rule to avoid projects that are not technically feasible, meaning true research projects. Nonetheless, the rhetoric of the commercial software industry, both aimed at unsophisticated investors and customers is the opposite. Terms like research, science, and research and development are used routinely to describe commercial software development activities. Many companies make statements that either explicitly claim or imply that the company has a large R&D group engaged in true research. Note that rhetoric aimed at sophisticated investors such as venture capitalists is often the opposite, which can be quite confusing.

Historically, the commercial software industry has relied heavily on government sponsored research programs such as the Defense Advanced Research Projects Agency (DARPA) and the National Aeronautics and Space Administration (NASA) for the true research in software. Many types of software and specific software products can be traced back to government sponsored research programs. Some well known examples include the Internet, originally a DARPA project, and the World Wide Web, which grew out of research projects at CERN and NCSA. Many other examples exist. Essentially all speech recognition software is derived from research sponsored by DARPA, especially projects at Carnegie-Mellon University. Nonetheless, industry rhetoric often invokes the image of private inventors in garages, the Wright brothers, and similar images of “free enterprise” and individual initiative. Typically, the putative inventor such as Tim Berners-Lee or Marc Andreessen is emphasized and the relevant government research program ignored or downplayed. Often there is little or no progress in commercial software if the relevant government research program is unable to make progress. This is most evident in pattern recognition and artificial intelligence, where progress has been very slow or non-existent.

Many government research programs are afflicted by a single “right way” that is pursued to the exclusion of all others. If this right way is good, then there is steady progress in the associated commercial software field. DARPA in particular relies upon periodic contests pitting different methods against one another. This has repeatedly resulted in a single approach that showed early promise taking over a field. A contest of this type during the 1970’s resulted in the so-called Hidden Markov Model (HMM) based speech recognition approach replacing essentially all speech recognition research on a global scale. Essentially all major speech recognition research groups, many directly funded by DARPA, pursue some variant of the HMM algorithm. Yet the performance of the HMM algorithm continues to be quite limited after 30+ years.

Most major commercial opportunities in complex algorithms require a company to fund and undertake genuine research, a difficult task that few companies understand. Artificial intelligence, speech recognition, cures for major diseases such as cancer or working fusion energy sources require substantial research.

Evaluating Technical Feasibility

In complex algorithms, technical feasibility generally means that a working prototype of the complex algorithm exists. In practice, working prototypes are often implemented in computer programming languages such as ANSI C, C++, or Java. Special algorithm research and development tools such as Matlab and Mathematica are also used. The working prototype may be slow, inefficient, or have other limitations, but it is or should be a proof of concept.

Seemingly, it should be easy to evaluate technical feasibility. Simply acquire and test the working prototype. Nonetheless, there is a long history of ostensibly sophisticated organizations and investors investing in complex algorithm projects that are not technically feasible, often apparently in the belief that the relevant problem had been solved. This has happened repeatedly with pattern recognition and artificial intelligence. It has also happened with various data compression algorithms.

For some reason, some organizations and investors do not even acquire and test the working prototypes. This is absolutely essential. It is often said that one should invest in people, not in ideas. Venture capitalists and other sophisticated investors often quote this platitude. However, the history of complex algorithms is filled with companies and projects with officers, directors, managers, chief scientists, and so forth with gold-plated resumes, impressive credentials, top degrees from top universities, and so forth that have flopped. The bottom line in complex algorithms is that the ideas count. If the complex algorithm doesn’t work, it doesn’t work. The company or project will fail for purely technical reasons. This is the lesson, for example, of the Pen Computing fad of the early 1990’s, GO, and Lernout and Hauspie (see below). This is also one of the lessons of the complex financial models involved in the current global financial crisis (2008). Due diligence requires a thorough, well-considered independent test and evaluation of the working prototype.

It can be difficult to evaluate technology demonstrations of complex algorithms. Often there are subtle and not so subtle ways to manipulate a technology demonstration. For example, in the early days (1995) of MPEG digital video on personal computers, MPEG software video player companies gave technology demonstrations on high-end personal computers with high-end video acceleration graphics cards. In pattern recognition, many present-day (2008) pattern recognition algorithms can achieve recognition rates in the range of 80-95% at least under certain controlled conditions. This seems high. In schools, 80-95% is usually a B or an A, a good grade. However, practical real-world pattern recognition often requires a recognition rate of 98 or 99% or even better under field conditions. In both examples, the technology demonstration can be misleading without demonstrably false statements.

Actual fraud can be difficult to detect and difficult to prove. It is important to conduct tests and evaluations of working prototypes independent of the purveyor or other interested parties who may have a reason to perpetrate a fraud. Complex algorithms can involve the potential for very large returns on investment—as noted above. Consequently, there is considerable motivation for fraud in some cases.

How Does Research Differ from Development?

Commercial software development is usually unpredictable. Software projects frequently involve unexpected problems and usually take substantially longer than planned. Nonetheless, technically feasible commercial software development projects are more predictable than true research projects. Often if one takes a conservative cost and schedule estimate and multiplies this by a factor of three to four, one gets the actual cost and schedule of the project. A common joke with a great deal of truth is: to get the real schedule multiply the official schedule by PI (3.14) for running around in a circle. Because the project is technically feasible it can certainly be completed. Massive cost and schedule overruns (such as factors of ten) can usually be explained by incompetence or severe political problems. True research is extremely unpredictable. Many true research projects simply fail. The researchers are unable to find the solution. For example, to date (2008), essentially all attempts to decipher human speech have failed in close to a century of attempts. Substantial research efforts at Bell Labs, MIT, and other institutions have failed to determine why certain sound spectra correspond to the different sounds in English and other languages. Even in successful research, estimates are often way off. For example, the mathematician Johannes Kepler made a bet in 1600 that he could determine the orbit of Mars in eight (8) days. His discovery of the elliptical orbit of Mars and other planets, one of the most important and difficult discoveries in scientific history, took five frustrating years in which every attempt to solve the problem failed until he found the answer in just a few days in 1605. This process in which long periods of little or no progress are punctuated by sudden unpredictable leaps forward is typical of true research, especially major scientific discoveries or technological inventions.

True research, especially major scientific discoveries or inventions, usually involves a very large amount of trial and error. Often, after many failures, there is a leap or leaps in which a new approach or concept is tried which unexpectedly solves the problem. Most major scientific discoveries or inventions took somewhere between five (5) and twenty (20) years. This is significantly longer than the time frame of typical commercial software industry and venture capital funded projects. In many cases, one is talking about five to twenty years of failure followed by a “breakthrough”, as in Kepler’s case.

In my experience, people involved in commercial software development are often unaware that they have little or no experience with true research. The misleading rhetoric of the computer software industry often leads people involved in commercial software development to think that they are engaged in the sort of true research conjured up by iconic names like Einstein or the Wright brothers. This undoubtedly leads to many bad decisions and frustrating experiences.

The Importance of Rapid Prototyping

History records many remarkable instances when an individual or small team succeeded in making a major scientific discovery or invention on a very small budget, sometimes beating far better funded competitors. Major scientific discoveries or inventions almost always involve a large amount of trial and error. Discoverers or inventors who managed to make a major discovery or invention on a shoe-string budget usually found a very fast, inexpensive way to perform the many trials and errors required to make a major discovery or invention.

For example, James Watt is remembered for inventing the separate condenser steam engine, a major conceptual leap that turned the steam engine from a niche device used in coal mining to a major driver of the industrial revolution. The Newcomen steam engines of Watt’s time were huge expensive house-sized engines. Watt however built and experimented with tiny scale models built from inexpensive wood, copper, and other materials. This enabled him to perform hundreds of trials and errors that led to the breakthrough concept of the separate condenser that radically improved the Newcomen steam engine.

Octave Chanute and the Wright brothers, his proteges, conducted research and development of gliders constructed of cheap wood and canvas. The gliders were flown at low altitude on soft sand beaches, first in Gary, Indiana near Chicago where Chanute lived, and later at Kitty Hawk in North Carolina. This meant that the inevitable damage from crashes was limited and easily repaired. The pilots did not die from the crashes as was common with other early would-be aviators. By delaying work on the expensive engines until last (they planned to buy a commercial off-the-shelf engine), they avoided the enormous cost involved in repairing or replacing an engine after each crash. This enabled Chanute and the Wright brothers to eventually succeed where better funded efforts such as Hiram Maxim and Samuel Langley failed.

Progress in aviation and rocketry today is quite slow, almost flat-lined since 1970, in part because the cost of a single trial, especially a new high performance engine, has become extremely high, easily in the millions if not billions of dollars per prototype engine and vehicle. In rocketry and other high performance engines, the prototype engine and vehicle are often destroyed during each trial. Internet entrepreneur Elon Musk of PayPal fame encountered this problem with his SpaceX startup as have many other Internet and software entrepreneurs attracted by the dream of space travel.

In algorithm research and development today, rapid prototyping tools such as Matlab and Mathematica (see below) speed up and reduce the cost of the many trials and errors required in true research. This is very important because the number of trials and errors is usually very large.

The Importance of Conceptual Analysis

Most major scientific discoveries and inventions usually involve a large amount of conceptual analysis expressed in words and pictures (often hundreds of thousands of words). It is common to find lengthy verbal discussions of the issues combined with rough sketches or drawings of concepts. For example, Octave Chanute wrote an entire book Progress in Flying Machines containing his lengthy verbal analysis of the problem of flight. This book outlines his successful research plan to develop working powered flight. It contains several rough drawings, as is common in major breakthroughs, and only a few brief calculations. The mathematician Johannes Kepler devoted much of his book, now known as New Astronomy, to a lengthy conceptual analysis of the problem of planetary orbits which was critical to his resolution of the problem.

At some point, these verbal analyses are refined into precise technical drawings in the case of mechanical inventions and specific mathematical expressions in the case of mathematical discoveries like Kepler’s. However, the verbal and visual analysis appears to be critical in many discoveries and inventions and usually comes first. It is likely that this sort of verbal and visual analysis will be essential to solve many problems such as artificial intelligence and pattern recognition.

Historically, this conceptual analysis was considered a part of philosophy. Much of the classical training in Greek philosophy and mathematics probably provided important training in this conceptual analysis. The discovery of new mathematical expressions of practical use strictly by the symbolic manipulation and the highly abstract thought favored by the famous mathematician David Hilbert and his school at the University of Gottingen in the early 20th century seems to be rare. This is specifically mentioned because Hilbert’s extremely abstract approach to higher mathematics has come to dominate mathematics and theoretical physics in the 20th century.

Some Famous Flops

Complex algorithm research and development is a treacherous area. There have been numerous flops and fiascoes over the years. It is easy to misjudge the technical feasibility of projects. There is a long history of exaggerated claims for complex algorithms that emulate aspects of human intelligence such as speech recognition. There has been enormous success in data compression over the last few decades. Nonetheless, there is a long history of exaggerated claims for advanced in data compression. Video and other data compression involves complex algorithms that are difficult to evaluate. Caveat emptor!

The Pen computing fad of the early 1990’s is an example of a famous flop. The most prominent of these firms was Jerry Kaplan’s GO, described in his book Startup. GO and similar firms’ business plans hinged on handwriting recognition, an unsolved problem in pattern recognition. Kaplan actually devotes only a few pages of his book to the handwriting recognition problem.

Another notorious example is the speech recognition firm Lernout and Hauspie. Lernout and Hauspie collapsed in a major financial scandal with court cases and allegations of massive fraud. Again, the success of Lernout and Hauspie’s business depended on the solution of the speech recognition problem, which remains largely unsolved even today.

Note that many apparently sophisticated investors invested many millions of dollars in both GO and Lernout and Hauspie, even though a modicum of research would have revealed the poor state of handwriting and speech recognition technology at the time.

Some Famous Successes

Video and audio compression is one of the most successful areas in complex algorithms. Technologies such as VideoCD, DVD, MP3, and BluRay all incorporate sophisticated audio and video compression algorithms.

A major breakthrough in video compression reached the market in 2003, embodied in H.264, Windows Media 10, Flash Video, and other video standards and products. Prior to 2003 the bitrate for usable, loosely VHS quality video was about one (1) megabit per second. In 2003, the new video technologies achieved a bit rate of around 275 Kilobits/second, often with close to DVD quality with proper tuning of the compression. This was a truly major advance, a rare technological leap forward. This enabled YouTube and other forms of Internet/web video over DSL connections.

The bottom line is that complex algorithm research and development can be done, but it is difficult.

Software Engineering

There are significant differences between software engineering for complex algorithm research and development and mainstream software development. As mentioned above, complex algorithms often involve a tight coupling between parts of the algorithm that makes development more difficult and tedious than most software development.

It is often easier to research and develop complex algorithms using tools such as Matlab, Mathematica, AXIOM, or Maxima (formerly known as MACSYMA). These are scripting languages similar to Python or PHP. They usually have implicit variable declaration and/or conversion. They are usually “weakly typed” languages and break many textbook rules of “good” software engineering. They include comprehensive, well-integrated libraries of mathematical, numerical, and statistical functions. They usually have a data type known variously as a list, vector, or matrix that represents sequences of numerical or symbolic data in a flexible way. These tools are sometimes referred to as computer algebra systems (CAS), although this is really only one subset of their features.

Adding Two Vectors in Mathematica

A = {1.0, 2.0, 3.0}; (* A is a Mathematica list *)
B = {1.1, 0.0, 4.0};
C = A + B
Out[1]={2.1, 2.0, 7.0}

Adding Two Vectors in C/C++

#include <iostream.h>
double A[3] = {1.0, 2.0, 3.0}; // A is a C++ array
double B[3] = {1.1, 0.0, 4.0};
double C[3];
int index;

for(index = 0; index <3; index++)
   C[index] = A[index] + B[index];

cout << “{“ << C[1] << “,” <<
C[2] << “,” << C[3] << “}” <<
endl;

Note that there is a vector class template in the C++ Standard Template Library (STL) with somewhat similar properties to the lists in Mathematica. The above comparison is for illustrative purposes. Even using the STL classes, it is usually much easier to research, develop, and test algorithms in these tools than using traditional compiled, strongly typed languages such as C/C++, Java, or <insert your favorite programming language here>. However, these tools are slow and require large amounts of memory. This is a significant drawback. Once an algorithm is developed, it is often necessary to convert the algorithm to a fast compiled language for performance reasons. This is easier if the target fast language has good libraries of mathematical, numerical, and statistical functions.

One can also research and develop algorithms directly in a fast programming language such as C/C++ or Java. This avoids conversion costs, speed, and memory issues. However, it is often much easier to do research and development using a tool such as Matlab or Mathematica.

The leading algorithm research and development tools are:

• MATLAB: Matlab is widely used in the commercial world, especially in digital signal processing.
• MATHEMATICA: Mathematica is widely used in government sponsored research and development and academic research. It has a following in Wall Street finance and economics.
• AXIOM: Axiom is free, open source, with a Berkeley style license. AXIOM was started in 1971 and has over 300 man years of work integrated into it.
• MAXIMA: Maxima is free, open-source, with a GNU license.

Many fast programming languages have been used for complex algorithms. The most popular are probably:

• ANSI C: ANSI C is almost universally available for all processors. It is simple, efficient, with small memory needs and high speed.
• C++: C++ is object-oriented. It often has larger memory needs than C and can be slower.
• Java: Java is compiled to byte-codes, but is approaching C/C++ in speed. It can be slower and less efficient. It can be easier to reverse engineer.

The dream algorithm R&D tool would be similar to Matlab or Mathematica but could be compiled to fast, efficient binaries similar to ANSI C and would be available for all platforms. An integrated GUI builder similar to Visual Basic and integrated network support would be helpful. The biggest single weakness of all kinds of scripting languages is that they are slow and cannot be compiled. For compute- intensive complex algorithms this can be a very significant problem. Of scripting languages, only Visual Basic 6 appears to have solved the problem of producing a compiler that can produce binary executables with similar performance to C/C++.

These algorithm research and development tools are not, of course, a substitute for thought, creativity, and the extensive conceptual analysis frequently required for major advances. Trial and error alone, without insight, rarely succeeds.

Conclusion

Complex algorithm research and development can be done successfully. Some great successes exist. Nonetheless, it is not easy and many things can go wrong. The project scope is significant. Project feasibility is difficult to assess. Genuine breakthroughs are unpredictable and take time. The return on investment for a home run can be five to thousands of times the original investment. There are great unrealized opportunities, most of which require genuine research.

About the Author

John F. McGowan, Ph.D. is a software developer, research scientist, and consultant. He works primarily in the area of complex algorithms that embody advanced mathematical and logical concepts, including speech recognition and video compression technologies. He has many years of experience developing software in Visual Basic, C++, and many other programming languages and environments. He has a Ph.D. in Physics from the University of Illinois at Urbana-Champaign and a B.S.in Physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11 (at) earthlink.net.

References

Some Complex Algorithms

• MPEG compression, one of the great success stories: http://www.chiariglione.org/mpeg/
• x264 is a free, opensource h.264 video encoder: http://www.videolan.org/developers/x264.html
• The Carnegie Mellon Sphinx Project, an open-source speech recognition engine: http://cmusphinx.sourceforge.net
• National Library of Medicine Insight Image Registration and Segmentation Toolkit: http://www.itk.org

Books and Articles

1. Startup: A Silicon Valley Adventure, by Jerry Kaplan, Houghton Mifflin Co, Boston, 1995, ISBN 0-140-25731-4
2. “How High-Tech Dream Shattered in Scandal at Lernout & Hauspie”, by Mark Maremont, Jesse Eisinger, and John Carreyrou, Wall Street Journal, December 7, 2000
3. New Astronomy (Nova Astronomia), by Johannes Kepler, Translated from the Latin original by William H. Donahue, Cambridge University Press, Cambridge, UK, 1992, ISBN 0-521-30131-9

This article was written by John F. McGowan, PhD. If you’d like to write for Math-Blog.com, please email us at submissions (at) math-blog.com.

It was “buzz worthy” for sure, and all the attention quickly attracted the interest of numerous scientists from many disciplines. As soon as the mathematicians, and particularly computer scientists, managed to get through its 1000+ pages, the first negative reviews began to pour in. Though, in all fairness, a few scientists had a little too much fun with this book and managed to showcase their comedic abilities by writing some of the most hilarious reviews known to humankind.

In this controversial best-seller, Stephen Wolfram primarily dissects the subject of cellular automata and its relevance to other scientific disciplines, in a systematic manner. It’s a book that covers a lot of ground and is arguably a remarkable piece of writing. Yet, the scientific community greeted the book with a fair dose of criticism.

So what went wrong? The main problem with A New Kind of Science is that it set very high expectations due to its author, title, and the numerous reminders of how important this material is, throughout the book.

The main accusations ranged from the book being called a display of Wolfram’s ego, to having very little “new” content, all the way to the more severe claims of not crediting other people’s work. For example, the idea of the universe as a cellular automaton was first presented by Konrad Zuse, so Wolfram’s “new” idea of a discrete, computable universe was anything but groundbreaking. On top of that, the most remarkable technical achievement revealed in this book was arguably the proof that the rule 110 cellular automaton is Turing complete. While this was conjectured by Wolfram, it was actually proven by his assistant Matthew Cook, who was refrained from publishing his results elsewhere by Wolfram’s lawyers.

It’s important to understand that, while perhaps not accepted as the breakthrough that Wolfram had hoped for, this book – and the methods for studying computational systems illustrated within it – is far from gibberish. Wolfram’s ambitious project failed in the eyes of the community due to the extremely high expectations that were set for this book. When you claim to have something radically new, you must be able to back that claim up in a convincing enough manner or else you’re bound to end up with egg on your face.

To be fair to Wolfram (for the few who are not familiar with his work) NKS is a controversial project, but he was already famous for having created the excellent program Mathematica (whose 7th version was recently released), one of the world’s most complete and advanced mathematical software.

Now Wolfram is at it again. According to his recent announcement, he is about to unleash something called WolframAlpha to the world, which combines both his work with Mathematica and NKS. In Wolfram’s own words:

I had two crucial ingredients: Mathematica and NKS. With Mathematica, I had a symbolic language to represent anything—as well as the algorithmic power to do any kind of computation. And with NKS, I had a paradigm for understanding how all sorts of complexity could arise from simple rules.

The project has been kept on the down-low for the past few years, while some of the brightest mathematicians and engineers employed by Wolfram Research, Inc. worked on it. It’s currently in private beta, but will go live in May of this year. From an initial glance, it would seem to be just another search engine a la Google.com. But is it? Not quite. It’s labeled as a “computational knowledge engine”, whose aim is to compute answers from the human knowledge available on the web. Whereas on Google you can search for strings and the results will be a series of relevant links, WolframAlpha will supposedly be able to parse and “understand” a query that’s inputted in English, and compute an answer based on the extensive knowledge stored in its system (assuming that a univocal answer exists). Conceptually speaking, it’s leaps and bounds more complex to get right than Google, which simply looks for matching strings and orders the results based on the popularity of the given keywords (For more information about the mathematics behind Google, read this book).

According to Nova Spivack, who had a chance to try out WolframAlpha, the service is able to compute factual answers to questions such as “What is the location of Timbuktu?”, “How many protons are in a hydrogen atom?,” “What was the average rainfall in Boston last year?,” “What is the 307th digit of Pi?,” “where is the ISS?” or “When was GOOG worth more than $300?”. This project has the potential to change the world as we know it, just like Google did. Several years ago Altavista was fine for most people’s search needs – or so we thought. It took Google to show us how much better off we could be search-wise, how much we needed Google, and ultimately how inadequate Altavista was. Unlike the case of Google and Altavista though, WolframAlpha would not replace Google, since the two services cover complimentary needs. Having access to a service that’s able to compute answers out of the chaos of the factual information that’s available to man would be a major breakthrough for humanity and computer science. And if an API (Application Programming Interface) were to become available, other developers would be able to tap into that with their applications.

Bold claims, high expectations. You understand why, two months away from experiencing something so potentially revolutionary, there is a lot of hype surrounding this project – but also major skepticism. For many this is A New Kind of Science all over again, especially since natural language processing and “computing knowledge” are extremely ambitious challenges in a realm where many have failed before. Pulling this one off would be a major accomplishment (that would dwarf Wolfram’s past achievements, including Mathematica), and, at long last, it would be the hard earned, practical validation of some of the methods and philosophies expressed in NKS by Wolfram.

I fully expect people to find bugs and have many simple questions, for which we will see bizarre answers. We’ll read blog posts about the whole thing and perhaps have a good laugh. But what interests me the most is whether, as Google did in the past, this new engine will be able to be practical and useful on an everyday level. Bugs are fair play and expected, but what we’re looking for here is a spark of true innovation thanks to the mathematical modelling of human knowledge.

I suspect that this engine will either have us in awe like Mathematica did, or leave us with mixed feelings – if not downright disappointment, like A New Kind of Science did for many. I can’t help but hope for the former, as I wait for my chance to try it out.

1. General Math Cheat Sheet (iPaper and other formats)
2. Elementary Algebra Cheat Sheet (PDF)
3. Trigonometry Cheat Sheet (PDF)
4. Calculus Cheat Sheet (PDF)
5. Derivatives and Integrals Cheat Sheet (PDF)
6. Laplace Transforms Cheat Sheet (PDF)
7. Abstract Algebra Cheat Sheet (PDF)
8. Probability Theory Cheat Sheet (PDF)
9. Matlab Cheat Sheet (PDF)
10. Mathematica Cheat Sheet (PDF)
11. Maple Cheat Sheet (PDF)
12. Maxima Cheat Sheet (HTML web page)
13. LaTeX Cheat Sheet (several formats)

And since most of us like to show our math pride off when out and about as well, Amazon sells this awesome Math Cheat Sheet T-shirt with formulas on both sides (Also available for Science and Engineering). How awesome is this?

Flexible, powerful, and easy to use, gnuplot is not only handy as a stand-alone program but can be used successfully by a variety of programming languages, including but not limited to Ruby, Python and Perl. It has also been adopted as a plotting engine by Open Source programs like Maxima and GNU Octave. The minimal effort required to learn gnuplot is therefore a very worthwhile endeavor. Being able to visualize things as you analyze data and explore mathematics, is a very useful aid.

gnuplot is cross-platform, and you can download it from SourceForge.

Once you have installed gnuplot, you can start the tool by simply running gnuplot from the shell. Getting started with the program is straightforward, given that the most basic functionalities are fairly intuitive. For example, if you wanted to plot the sin(x) function, you could run:

gnuplot> plot sin(x)

gnuplot will automatically decide for you what portions of the function should be visualized (in our case between -10 and 10 on the x axis). If you want to overwrite this, you can specify otherwise, as shown below:

gnuplot> plot [-pi:pi] sin(x)

Similarly, you can customize the plotting, by specifying a range on the y axis:

gnuplot> plot [] [-0.5:0.5] sin(x)

It is also possible to plot several functions at once:

gnuplot> plot [-2:2] x, x**2, x**3

gnuplot can draw all sorts of functions and graphics. For example, the following plots two surfaces:

gnuplot> splot x**2+y**2, x**2-y**2

What you see plotted above, is based on a sensible set of defaults, but gnuplot offers countless options to customize the appearance of your graphics.

gnuplot isn’t just useful for plotting mathematical functions. I often use it for plotting data that is stored in tabular format in simple text formats. For example, assume that you have collected the following data on Deaths by Major Causes in the US, from a website:

stats.txt

That’s a big chunk of data, but it’s not very easy to analyze. It’d be nice to be able to visualize it. Assuming you saved it in a stats.txt file (you can include or remove the first few lines of comments) from gnuplot you can run the following:

gnuplot> plot "stats.txt"

That’s a meaningless mess. But gnuplot is very flexible, so we can easily do much better. Let’s use the first column for x values, and each of the remaining columns as a curve of its own by running:

gnuplot> plot "stats.txt" using 1:2 title "Heart disease" with lines, "stats.txt" using 1:3 title "Cancer" with lines, "stats.txt" using 1:4 title "Cerebro-vascular diseases" with lines, "stats.txt" using 1:5 title "Lower respiratory diseases" with lines, "stats.txt" using 1:6 title "Diabetes mellitus" with lines, "stats.txt" using 1:7 title "Influenza and pneumonia" with lines, "stats.txt" using 1:8 title "Chronic liver disease" with lines, "stats.txt" using 1:9 title "Accidents" with lines, "stats.txt" using 1:10 title "Suicide" with lines, "stats.txt" using 1:11 title "Homicide" with lines

Or it’s abbreviated version:

gnuplot> plot "stats.txt" u 1:2 title "Heart disease" w l, "" u 1:3 title "Cancer" w l, "" u 1:4 title "Cerebro-vascular diseases" w l, "" u 1:5 title "Lower respiratory diseases" w l, "" u 1:6 title "Diabetes mellitus" w l, "" u 1:7 title "Influenza and pneumonia" w l, "" u 1:8 title "Chronic liver disease" w l, "" u 1:9 title "Accidents" w l, "" u 1:10 title "Suicide" w l, "" u 1:11 title "Homicide" w l

Much better! Of course, the command above has a bit of repetition in it, but it’s justified by the flexibility of being able to pull data from different files if required, and plot some data with lines and others without (or in different styles) if we want to. To improve the appearance and legibility of the chart, let’s add a grid:

gnuplot> set grid
gnuplot> replot

Nice! And notice how the replot command runs the last plotting directive for us, saving us some typing (or scrolling through the previous commands). To improve this further, we can also set the labels for the axis and a title for the whole chart:

gnuplot> set xlabel "Years"
gnuplot> set ylabel "Death rates per 100,000 people"
gnuplot> set title "Deaths by Major Causes, 1960–2005"
gnuplot> replot

(Click here for a larger version)

Effortlessly, we went from having some data in a text file, to a meaningful, and professional looking chart that could be used in an article or a book. Not bad. You can save it in your favorite format, by setting the output file name and the terminal type. In the example below, I specify the font type, size and its location on my Mac OS X system, but it’s entirely optional. If missing, the default fonts apply.

gnuplot> set output "mychart.png"
gnuplot> set terminal png font "/Library/Fonts/Times New Roman.ttf, 11"
gnuplot> replot

This very short introduction should you give you a glimpse into what gnuplot can do for you, and even get you started with the tool. But gnuplot is much more than this and infinitely customizable, so you really may want to consider learning more about it. Unfortunately, despite being widely used for more than a decade, gnuplot has never had a book published about it.

Thankfully, Manning Publications will be putting out a book called Gnuplot in Action in October. The good news is that you don’t have to wait for the dead tree version, you can purchase it today through the Manning Early Access Program (MEAP). 12 out of the 15 chapters are already available, so the book is pretty much complete. If you’re interested in making the best out of gnuplot, this book is a must have. You don’t have to be a mathematician or a programmer to follow along, and it’s so much more than a bunch of options for customizing your graphics. The book guides you through the best techniques for taking advantage of the tool in several common scenarios. By reading, this easy to follow book, you’ll be able to generate colorful, nifty plots and have good mastery of the tool in a very short amount of time.

An Introduction to the Field of Root-finding

Given a function which is defined in terms of one or more independent variables, the roots (also called zeros) of the equation are the values of the independent variable(s) for which the function equals :

Note that, in general, the z values are complex numbers, comprised of real and imaginary components (indicated by the ): .

Consider the following equation:

The values of and which satisfy this equation may not be possible by analytical methods, so the equation would be rearranged into the following form:

which is of the form

Once in this form (the standard form), solving the original equation becomes a matter of finding the and values for which f equals , and well-developed fields of mathematics and computer science provide several root-finding techniques for solving such a problem. Note that since the theme of this article is polynomial root-finding, further examples will focus on single-variable equations, specifically, polynomials.

Analytical Root-finding Techniques

Consider the following quadratic equation:

This equation is a second-degree polynomial–the highest power applied to the independent variable is 2. Consequently, this equation has two roots; an n-degree polynomial equation has n roots.

Because the equation is a simple polynomial, a first approach to finding its zeros might be to put the equation into the form and make a few educated guesses; a person could quickly determine that (a strictly real result) is one root of this equation. This root could then be divided out of the original polynomial

to yield the second root: .

In this simple example, the two roots were found easily; the second root was found immediately after the first one was known. However, in most real-world problems the roots of a quadratic are not found so easily. Furthermore, in problems where the original polynomial is of degree greater than two, when one root is found, the other roots do not follow immediately — more sophisticated techniques are required.

Among the techniques available are analytical ones, in other words, techniques that yield explicit, algebraic results. For example, in the case of a quadratic equation, explicit expressions for the two roots are possible via the Quadratic Formula: once a quadratic equation is arranged into the standard form

(a, b and c are known constants), the two roots are found via the Quadratic Formula:

The quantity within the square root sign is called the discriminant and determines whether the solution is complex or strictly real. If the discriminant is less than , the numerator contains an imaginary component and the roots are, therefore, complex. If the discriminant is greater than or equal to , the solutions are real. In fact, if the discriminant is , the two roots are real and equal:

In addition to the Quadratic Formula for quadratic equations, analogous formulae exist for polynomials of degree three and four. However, for polynomials of degree five and higher, analytical solutions are not possible (except in special cases), only numerical solutions are possible.

Numerical Root-finding Techniques

For the multitude of real-world problems that are not amenable to an analytical solution, numerical root-finding is a well-developed field offering a wide variety of tools from which to choose; many computer programs written for the computation of roots are based on algorithms that make them applicable to the computation of roots of functions in addition to polynomials, and virtually all of them employ an iterative approach that terminates once the desired degree of tolerance has been achieved. For example, the bisection method is a simple and reliable method for computing roots of a function when they are known ahead of time to be real only. The algorithm starts with the assumption that a zero is somewhere on a user-supplied interval [a,b]. In other words, the function value f(a) is of the opposite sign of f(b) – in going from a to b, f either goes from being positive to being negative, or it goes from being negative to being positive. A point, m, in the middle of the interval [a,b] is then selected. The function is evaluated at point m: f(m). If the sign of f(m) is the same as the sign of f(a), the desired zero is not in the interval [a,m]. In this case, the interval is cut in half and the new interval becomes [m,b]. On the other hand, if the sign of f(m) is not the same as the sign of f(a), the desired zero is in the interval [a,m]. In this case, the interval is cut in half and the new interval becomes [a,m]. This process is repeated until either the interval converges to a desired tolerance or a value of f is found that is within an acceptable tolerance, in which case the value of z at this point is the value returned by the program as the root.

In addition to the bisection method, more elegant–and faster converging–techniques are available: the False Position method, Brent’s method, the Newton-Raphson method, the secant method, and others. The Newton-Raphson method uses the function and its derivative to quickly converge to a solution. This method is good for situations in which the derivatives are either known or can be calculated with a low computational cost. In other situations, the cost of computing derivatives may be too high. The secant method, by comparison, does not require the use of derivatives, but the calculation of each iterate requires the use of the two previous iterates and does not converge to a solution as quickly as the Newton-Raphson method. In some situations, this method, too, may be deemed impractical. Usually, when some thought is given to a problem, a particular technique can be applied to it that is more appropriate than another. In fact, for some problems, one technique may fail to find a solution at all, whereas another technique will succeed. For other problems, several techniques may, indeed, be able to solve the problem and the numerical analyst may select the one that is simply more computationally efficient.

The Jenkins-Traub Algorithm

The field of numerical root-finding is so well-developed that the use of a good-quality numerical program is recommended — when numerical results are sought — even when an analytical solution to a problem is known. In other words, even though an analytical solution to, say, the quartic equation exists, writing a computer program that simply implements the textbook formula is not recommended; computer round-off errors often render results of such programs meaningless. The use of a robust numerical program, based upon sound theory and an excellent algorithm, and coded to thoroughly deal with computer round-off errors, is the recommended action. The Jenkins-Traub Algorithm is such an algorithm; it is a three-stage, extremely effective, globally convergent algorithm designed specifically for computing the roots of polynomials.

Stage One is the “No Shift” stage; the main purpose of this stage is to accentuate the smaller zeros. The search for a zero is started by taking an initial guess of for a fixed number, M, of iterations (M is usually assigned the value 5 on the basis of numerical experience⁽¹⁾).

Stage Two is the “Fixed Shift” stage, the purpose of this stage is to separate zeros of equal or almost equal magnitude. As a starting point in this stage, the following value is used:

is a lower bound on the magnitudes of the probable zeros in the cluster. could be taken at random, since the cluster could be anywhere in the complex plane; however, in practice is usually initialized to 49°, putting s near the middle of the first quadrant of the complex plane. After a certain number of iterations, if s does not converge to a root, s is assigned a new value by increasing by 94°. Repeated attempts would have the search for a root start with points in all four quadrants of the complex plane until the search is returned to the first quadrant. Should the search, indeed, return to the first quadrant, successive cycles start at points 16° away from the starting point of the preceding cycle.

Stage Three is the “Variable Shift” stage, which is terminated when the computed value of the polynomial at a possible zero is less than or equal to a specified bound.

In addition to the three fundamental stages around which it was developed, the Jenkins-Traub Algorithm incorporates several other techniques for making it as effective as possible. One of those techniques is deflation of the polynomial by synthetic division each time a root is found. Consider the following monic polynomial:

The are known constants and, in general, are complex. Now say the root () has been found. Synthetic division would be employed to divide that root out of the original polynomial:

to yield

The are new constants. The root-finding process is then repeated on this new–simpler–polynomial. As each root is found, the polynomial becomes successively simpler and each successive iteration of the algorithm involves, in general, fewer computations.

For polynomials whose coefficients are real only, when a complex root is found (), an additional benefit arises: that root’s complex conjugate is also a root (). In other words, two roots are computed — and the polynomial can be deflated by two degrees — in a single iteration of the algorithm. Furthermore, this deflation involves real only, rather than complex, operations; the product of these two roots is a real quadratic equation:

In this case, synthetic division is employed on the polynomial as follows:

In fact, for computing roots of polynomials which have only real coefficients, a modified version of the Jenkins-Traub Algorithm has been written which incorporates several features that take advantage of the characteristics of real-only polynomials to yield significant decreases in program execution time; “If the complex and real algorithms are applied to the same real polynomial, the real algorithm is about four times as fast.”⁽⁴⁾

The Jenkins-Traub Algorithm not only deflates the polynomial as roots are computed, it computes the roots roughly in order of increasing magnitude. This approach is taken because deflation of a polynomial can be unstable unless done by factors in order of increasing magnitude. By starting the search for roots with an initial guess of , as is done in Stage One, roots are, indeed, computed roughly in order of increasing magnitude, the factors by which the polynomial is successively deflated are roughly in order of increasing magnitude — the deflation of the polynomial is made quite stable.

As a quick check of the effectiveness of the Jenkins-Traub Algorithm, consider a contrived numerical example:

The roots of this polynomial are very close and should test how effectively the algorithm discerns near-multiple roots:

1.001
0.998
1.00002
0.99999

In addition, the polynomial is real-only so that readers may confirm the results with a free, ready-to-use, online Javascript program (URL given below).

Expanded, the test polynomial becomes

and the solutions computed by the Javascript root-finder follow:

0.9980051486669109
0.9998618452719573
1.0001539207016332
1.0009890853594987

The program, in fact, did a very good job:

1. no errors were returned,
2. no erroneous imaginary components were found,
3. four distinct roots were computed, and
4. the computed roots are extremely close to the exact roots (round-off errors are unavoidable).

Conclusion

Since its introduction in 1969 (in Michael Jenkins’ PhD thesis) the Jenkins-Traub Algorithm has been rigorously tested and has proven itself as an excellent algorithm for the computation of the roots of polynomials. It is extremely robust, is globally convergent for any distribution of zeros, and converges at a better than quadratic rate. Attesting to its high quality is the fact that source code for programs (written in FORTRAN) based on the algorithm have been posted on the NETLIB site, a repository of high-quality numerical programs; the complex coefficient version is posted as CPOLY (http://www.netlib.org/toms/419) and a real coefficient only version has been posted as RPOLY (http://www.netlib.org/toms/493). Translations into C++ and Javascript are also available; for example, Polynomial Root-finder (http://www.akiti.ca/PolyRootRe.html) is a Javascript translation of RPOLY and has been successfully tested with data published in Jenkins’ PhD thesis. In addition, the algorithm is incorporated in commercial software products (i.e. Mathematica). The proven effectiveness of the Jenkins-Traub Algorithm and its wide-spread distribution make it an extremely important tool in the toolbox of numerical analysts.

References

1. “A First Course in Numerical Analysis: Second Edition”
   Anthony Ralston and Philip Rabinowitz
   McGraw-Hill Book Company
   New York
   1978
2. “Numerical Methods and Software”
   David Kahaner, Cleve Moler, Stephen Nash
   Prentice-Hall, Inc.
   Englewood Cliffs, New Jersey
   1989
3. “Numerical Recipes. The Art of Scientific Computing.”
   William H. Press, Brian P. Flannery, Saul A. Teukolsky, and William T. Vetterling
   Cambridge University Press
   New York
   1986
   (3rd edition now available.)
4. Wikipedia
   Jenkins-Traub Algorithm for Polynomial Zeros
5. Jenkins, M.A. Three-stage variable-shift iterations for the solution of polynomial equations with a posteriori error bounds for the zeros. Diss., Rep. CS 138, Comput. Sci.
   Dep., Stanford U., Stanford, Cal., 1969.

This article was written by David Binner. You can visit his math site at http://akiti.ca. If you’d like to write for Math-Blog.com, please email us at .