Math-Blog » Software

Creating Cartoon Voices with Math

John F. McGowan, Ph.D. — Mon, 15 Aug 2011 11:00:33 +0000

Have you ever wanted to create a humorous or entertaining voice like a cartoon character’s voice for a get-well video, a Valentine’s video, the narration for a DVD of home videos, an advertisement for your business or some other application? This article tells how to create cartoon voices using mathematics to shift the pitch of normal voices. The article includes the Octave source code for an Octave function chipmunk that applies pitch shifting to audio.

The standard audio pitch shifting incorporated in many commonly used audio editors such as the free open-source Audacity editor is presented in detail. The article also shows the results of using a more sophisticated algorithm that produces a more natural sounding pitch-shifted voice similar to the voice of the famous cartoon character Mickey Mouse.

One of the basic concepts and methods of signal and speech processing is the Fourier transform, named after the French mathematician and physicist Joseph Fourier. The basic concept is that any real function can be represented as the sum of the trigonometric sine and cosine functions. For example, a function defined on the region can be expanded as the sum of sines and cosines:

where the coefficients and are known as Fourier coefficients. This is a continuous Fourier Transform.

There is a discrete version of the Fourier Transform, often used in digital signal processing:

where is the index of an array of discrete values such as audio samples, is the value of the th audio sample, is the index of the discrete Fourier coefficients and is the number of discrete values such as the number of audio samples in an audio “frame”. The index is essentially the frequency of the Fourier component. This version of the discrete Fourier Transform uses the mathematical identity:

where

to combine the cosine and sine function components into complex functions and numbers.

In audio signal processing such as speech or music, the Fourier Transform has a straightforward meaning. The sound is broken up into a combination of frequency components. In most instrumental music, this is very simple. The music is a collection of notes or tones with specific frequencies. Percussion instruments and certain other instruments can produce more complex sounds with many frequency components. A spectrogram of a signal such as speech or music shows time on the horizontal axis and the strength of the frequency component on the vertical axis. This is the spectrogram of a pure 100 Hertz (cycles per second) tone:

Spectrogram of 100 Hz Tone

The spectrogram is generated using the specgram function in the Octave signal signal processing package by dividing the signal into a series of overlapping audio frames. Overlapping audio frames are frequently used to achieve better time resolution during signal processing in the Fourier domain. Each audio frame is windowed using the Hanning window to reduce aliasing effects.

The Fourier transform is applied to each windowed audio frame, giving a series of frequency components, which are displayed on the vertical dimension of the spectrogram. Each frequency component is a bin in frequency covering a frequency range equal to the audio sample rate divided by the number of samples in the audio frame. This frequency bin size or frequency resolution of the Fourier transform is about 20 Hz in the spectrogram above (44100 samples per second/2048 samples in an audio frame = 21.533 cycles per second). Because the 100 Hz tone in the example is not perfectly centered in the frequency bin spanning 100 Hz, the tone spreads out in the spectrogram, contributing to other bins as can be seen above. This is a limitation of the discrete Fourier transform which can lead to problems with signal processing such as pitch shifting.

Speech has a much more complex structure than a pure tone. In fact, the structure of speech remains poorly understood which is why current (2011) speech recognition systems perform poorly in realistic field conditions compared to human beings. This spectrogram shows the structure of the introduction to United States President Barack Obama‘s April 2, 2011 speech on the energy crisis: “Hello everybody. I’m speaking to you today from a UPS customer center in Landover, Maryland where I came to talk about an issue that is affecting families and businesses just like this one — the rising price of gas and what we can…”.

President Obama on the Rising Price of Gas

The spectrogram below shows the region from 0 to 600 cycles per second (Hertz). One can see a series of bands in the spectrogram. These bands are located at integer multiples (1, 2, 3, …) of the lowest frequency band, which is often referred to as F0 in the scholarly speech literature. The bands are known as the harmonics. F0 is known as the fundamental frequency. This is the frequency of vibration of the glottis which provides the driving sound for speech and is located in the throat. The glottis vibrates at frequencies ranging from as low as 80 cycles per second (Hertz) in some men to as high as 400 cycles per second (Hertz) in some women and children. This fundamental frequency appears to be loosely correlated with the height of the speaker, higher for short speakers such as children and lower for taller women and men.

The fundamental frequency F0 fluctuates in a rhythmic pattern that is not well understood as people speak. In some languages such as Mandarin Chinese, the changing pitch conveys meaning; a word with rising pitch has a different meaning from an otherwise identical word with falling pitch. In English, a rising pitch at the end of a phrase or sentence indicates that a question is being asked. “The chair.” is pronounced with falling pitch whereas “The chair?” is pronounced with a rising pitch at the end. It is difficult and even sometimes impossible to understand English if the rhythmic pattern of the fundamental frequency or pitch is abnormal.

President Obama on the Rising Price of Gas (to 600 CPS)

This spectrogram shows President Dwight David Eisenhower saying “in the councils of government we must guard against the acquisition of unwarranted influence, whether sought or unsought, by the military industrial complex” from his Farewell Address, January 17, 1961, probably his most famous phrase and his most famous speech today.

Eisenhower on the Military Industrial Complex

This spectrogram shows the spectrogram in the range 0 to 600 Hertz (cycles per second). Again, one can easily see the repeating bands.

Eisenhower on the Military Industrial Complex (to 600 CPS)

Human beings perceive something which we call “pitch” in English which appears closely related to or identical to the center frequency of the F0 band in the spectrogram. The F0 band will be higher in higher pitched speakers such as many women and most children. Both President Obama and President Eisenhower have similar pitches, varying between 200 and 75 Hertz with an average of about 150 Hertz. Nonetheless, their voices sound very different. The F0 band can be as low as 70 or 80 Hertz (cycles per second) in a few speakers. Former California governor and actor Arnold Schwarzenegger used an extremely low pitched voice while playing the Terminator, his most famous role.

In general, low pitched voices tend to convey seriousness and sometimes menace whereas high pitched voices tend to convey less seriousness, although there are exceptions. The voice of the genocidal Daleks in the BBC’s Dr. Who series is both high pitched and menacing at the same time. Cartoon style voices can be created by shifting the pitch of normal speakers. This has been done for the Alvin and the Chipmunks characters created by Ross Bagdasarian Sr.. It is probable that some form of pitch shifting has been used over the years to create some of the voices of the Daleks on Dr. Who. Some robot voices have probably been created by combining pitch shifting with other audio effects.

Traditional Pitch Shifting

Pitch shifting predates the digital era. In the analog audio era, one could shift the pitch of a speaker by playing a record or tape faster or slower than normal. This shifts the pitch but also changes the tempo — speed or rate of speaking — as well. One can achieve a pure pitch shift by, for example, recording a voice performer speaking at half normal speed and then playing the recording back at twice the normal rate. In this case, the pitch will be shifted up by a factor of two and the tempo or rate of speaking will be normal. One can create the Alvin and the Chipmunks high pitched voice in this way using analog tapes or records. One can also create lower pitched voices by appropriately combining the tempo of the original voice and the playback rate of the recording. Although these voices are easily understandable, they have artificial, electronic qualities not found in normal low or high pitched speakers or voice performers intentionally creating a low or high pitched voice. The voice of Walt Disney’s Mickey Mouse was performed by a series of voice artists starting with Walt Disney himself. This high pitched voice sounds much more natural than the Alvin and the Chipmunks voice.

In digital audio, it is possible to shift the pitch of the voice without changing the tempo of the speech. This can be done by manipulating the Fourier transform of the speech, the spectrogram, and converting back to the “time domain,” the actual audio samples. One can simply shift the Fourier components from their original frequency bin in the spectrogram to an appropriate higher or lower frequency bin. For example, if a Fourier component is in the 100 Hz bin, one shifts this Fourier component value to the 200 Hz bin to double the pitch. This must be done for each and every non-zero Fourier component. In general, this will produce a recognizable pitch shifted voice. If the Fourier components are not centered in each bin, which is normally the situation, this pitch shifted voice will have an annoying beat or modulation. It is necessary to perform some additional mathematical acrobatics to compensate for these effects to produce a relatively smooth pitch shifted voice similar to the output of the analog processing described above.

This video is President Obama’s original introduction from his April 2, 2011 speech on the energy crisis. Click on the images below to download or play the videos.

This video is President Obama speaking with his pitch doubled by shifting the Fourier components but without the mathematical acrobatics to compensate for un-centered frequency components:

This video is President Obama speaking with a chipmunked voice; his pitch has been doubled.

This video is President Obama speaking with a deep voice; his pitch has been reduced to seventy percent of normal.

Octave is a free open-source numerical programming environment that is mostly compatible with MATLAB. The Octave source code below, the Octave function chipmunk, implements the standard pitch shifting algorithm in widespread use. The Octave code requires both Octave and the Octave Forge signal signal processing package for the specgram function which computes the spectrogram of the signal.

The videos in this article were created by downloading the original MPEG-4 videos from the White House web site and splitting the audio and video into a MS WAVE file and a sequence of JPEG still images using the FFMPEG utility. Presidential speeches and video are in the public domain in the United States. The original still images were reduced in size by half using the ImageMagick convert utility. The audio was pitch shifted in Octave using the chipmunk function below. The new audio and video were recombined into the MPEG-4 videos in this article by again using the FFMPEG utility. Variants of this pitch shifting algorithm can be found in many programs including the widely used free open-source Audacity audio editor (the Audacity pitch shifting algorithm may be slightly different from the algorithm implemented below):

function [ofilename, new_phase, output] = chipmunk(filename, pitchShift, fftSize, numberOverlaps, thresholdFactor)
% [ofilename, new_phase, output] = chipmunk(filename [,pitchShift , fftSize, numberOverlaps, thresholdFactor]);
%
% chipmunk audio effect (as in Alvin and the Chipmunks)
%
% ofilename -- name of output file with pitch shifted audio
% new_phase -- the recomputed phases for the pitch shift audio (for debugging)
% output -- the pitch shifted audio samples
%
% arguments:
%
% filename -- input file name (MS Wave audio file)
% pitchShift -- frequency/pitch shift (default=2.0)
% fftSize -- size of FFT (default = 2048)
% numberOverlaps -- number of overlaps (default = 4)
% thresholdFactor -- threshold factor for zeroing silence frames
%
% $Id: chipmunk.m 1.44 2011/08/04 01:25:35 default Exp default $
% (C) 2011 John F. McGowan, Ph.D.
% E-Mail: jmcgowan11@earthlink.net
% Web: http://www.jmcgowan.com/
%

if nargin < 2
	pitchShift = 2.0; % frequency shift
end
nPitchShift = uint32(pitchShift*100); % to write output file

if nargin < 3
	fftSize = 2048; % size of audio blocks/FFT size
end

if nargin < 4
	numberOverlaps = 4; % number of overlaps
end

if nargin < 5
	thresholdFactor = 0.002;
end

printf("pitchShift: %f fftSize: %d numberOverlaps: %d thresholdFactor: %f\n", pitchShift, fftSize, numberOverlaps, thresholdFactor);
fflush(stdout);

stepSize = fftSize/numberOverlaps;
phaseShift = 2.0*pi*(stepSize/fftSize);

printf("loading %s\n", filename);
fflush(stdout);

result = char(strsplit(filename, '.'));
filestem = result(1,:);
ext = sprintf("_oct_%d_%d_%d.wav", nPitchShift, fftSize, numberOverlaps);
ofilename = [filestem ext];

[data, sampleRate, bits] = wavread(filename);

freq_resolution = sampleRate / fftSize; % frequency resolution = sample rate / fft size

if columns(data) > 1
	raw_data = data(:,1); % input is stereo with 2 channels in 2 columns of array
else
	raw_data = data; % mono sound input
end
data = [];
clear data; % free memory

mx_input = max(abs(raw_data(:)));

printf("applying fft\n");
fflush(stdout);
%spectrogram = fft(spectrogram);

overlap = fftSize - stepSize;
printf("stepSize: %d overlap is %d\n", stepSize, overlap);
fflush(stdout);

nsamples = length(raw_data);

% hanning window
window = hanning(fftSize); % window the output
window = (numel(window)/sum(window(:)) )*window; % normalize the window

% use Octave signal package specgram function to apply fft to windowed overlapping frames
% [] indicates default window (hanning)
%
[spectrogram, f, t] = specgram(raw_data, fftSize, sampleRate, window, overlap);

printf("spectrogram has dimensions %d %d\n", rows(spectrogram), columns(spectrogram));
fflush(stdout);

% free memory
raw_data = [];
clear raw_data;

intensity = dot(spectrogram, spectrogram, 1); % each column is an audio frame
max_intensity = max(intensity(:));
threshold = thresholdFactor*max_intensity;

speech_frames = intensity > threshold;

printf("speech_frames has dimensions: %d %d \n", rows(speech_frames), columns(speech_frames));
fflush(stdout);

printf("zeroing silence frames...\n");
fflush(stdout);

speech_frames = repmat(speech_frames,rows(spectrogram), 1);

spectrogram = spectrogram .* speech_frames; 

printf("dimensions spectrogram are now: %d %d \n", rows(spectrogram), columns(spectrogram));
fflush(stdout);

printf("computing phase...\n");
fflush(stdout);

% spectrogram is half-array without duplicate fft coefficients
% 1:fftSize/2 rows, number time steps columns
% each row is an fft coefficient
%
magn = 2.*abs( spectrogram ); % magnitude of fft coefficients
phase = arg( spectrogram ); % phase of fft coefficients

previous_phase = zeros(size(phase));
previous_phase(:,2:end) = phase(:,1:end-1);

phaseShifts = (0:(fftSize/2)-1)*phaseShift; % expected phase shift if frequency component is centered in bin
phaseShifts = repmat(phaseShifts', 1, columns(phase));

spec_buf = phase - previous_phase; % change in phase from previous time step
spec_buf = spec_buf - phaseShifts; % difference between change in phase and expected phase change
         % if frequency component is centered in frequency bin

printf("computing phase adjustment\n");
fflush(stdout);
									% handle mapping to -pi to pi range of atan2/arg (below)
phase_adjust = uint32(spec_buf./pi); % 0 if spec_buf between -pi and pi
phase_adjust = phase_adjust + ((phase_adjust >= 0).*(2) - 1).*bitand(phase_adjust,1);

spec_buf = spec_buf - pi*double(phase_adjust);
spec_buf = numberOverlaps*spec_buf./(2*pi);

printf("computing corrected frequencies\n");
fflush(stdout);
% compute corrected frequency
frequencies = repmat(f',1,columns(spectrogram)); % f is row vector when returned by specgram

spec_buf = frequencies + spec_buf*freq_resolution;

corrected_freq = spec_buf;

printf("applying frequency shift\n");
fflush(stdout);

shifted_magn = zeros(size(magn));
shifted_freq = zeros(size(corrected_freq));

oldTime = time;
for k = 1:fftSize/2
	ind = uint32((k-1)*pitchShift) + 1;
	if (ind <= fftSize/2)
		shifted_magn(ind,:) += magn(k,:);
		shifted_freq(ind,:) = corrected_freq(k,:) * pitchShift;
	end
	newTime = time;
	deltaTime = newTime - oldTime;
	if (deltaTime > 1)
		pct = (k / fftSize)*100.0; % percent progress
		printf("frequency shift: processed %3.1f%% %d/%d\n", pct, k, fftSize);
		fflush(stdout);
		oldTime = time;
	end % end if
end

%shifted_freq = corrected_freq * pitchShift;

% now convert from mag and freq to mag and phase
%
printf("computing new phase\n");
fflush(stdout);

spec_buf = zeros(size(spectrogram)); % make sure start with zeros

printf("new phase: assigning shifted frequencies\n");
fflush(stdout);

spec_buf(2:end,:) = shifted_freq(2:end,:);

printf("new phase: subtracting center frequencies\n");
fflush(stdout);

spec_buf(2:end,:) = spec_buf(2:end,:) - (frequencies(2:end,:) );

printf("new phase: dividing by frequency resolution\n");
fflush(stdout);

spec_buf(2:end,:) /= freq_resolution;

printf("new phase: adjusting for overlap\n");
fflush(stdout);

spec_buf(2:end,:) = 2.*pi*spec_buf(2:end,:)/numberOverlaps;

printf("new phase: computing delta phase\n");
fflush(stdout);

delta_phase = spec_buf + phaseShifts;

%delta_phase = phaseShifts;

new_phase = delta_phase;

printf("new phase: adding delta phase\n");
fflush(stdout);

%new_phase = spec_buf;
new_phase = zeros(size(spec_buf));
% % %new_phase(:,1) = spec_buf(:,1);
% % %dc coefficient has no phase (always a non-negative real)
oldTime = time;
ncols = columns(spec_buf);
for i = 2:ncols
	new_phase(2:end,i) = new_phase(2:end,i-1) + delta_phase(2:end,i-1);
	newTime = time;
	deltaTime = newTime - oldTime;
	if (deltaTime > 1)
		pct = (i / ncols)*100.0; % percent progress
		printf("new phase: processed %3.1f%% %d/%d\n", pct, k, fftSize);
		fflush(stdout);
		oldTime = time;
	end % end if
end

spec_buf = [];
clear spec_buf; % free memory

new_spectrogram = zeros(fftSize, columns(spectrogram)); % allocate full fft array for inverse fft

new_spectrogram(1,:) = shifted_magn(1,:); % dc coefficient
new_spectrogram(2:fftSize/2,:) = shifted_magn(2:end,:).*cos(new_phase(2:end,:)) + i*shifted_magn(2:end,:).*sin(new_phase(2:end,:));

new_spectrogram(fftSize/2 + 2:end,:) = conj(flipud(new_spectrogram(2:fftSize/2,:))); % reflect fft coefficients

spectrogram = [];
clear spectrogram;

% INVERSE FFT
%
printf("applying inverse fft\n");
fflush(stdout);

new_data = real(ifft(new_spectrogram))/fftSize; 

printf("dimensions new_data are %d %d\n", rows(new_data), columns(new_data));
fflush(stdout);

new_spectrogram = [];
clear new_spectrogram;

% each column is an audio frame which may overlap with previous audio frame by overlap samples
%

iframe = 1; % start at frame 1
it = 1; % start at first sample of output
output = zeros(nsamples,1); % all rows, 1 column

printf("applying overlap and add...\n");
fflush(stdout);

while( (it+fftSize-1) < nsamples)
	update = (new_data(:,iframe).*window)/numberOverlaps; % row of audio data
	output(it:it+fftSize-1) = output(it:it+fftSize-1) + update(1:fftSize);
	it = it + stepSize; % advance to next time
	iframe = iframe + 1; % advance to next audio frame (column of new_data)
end % while

new_data = [];
clear new_data;

mx = max(abs(output(:)));
%mean = sum(abs(output(:)))/numel(output);

if mx > 1.0
	scale_factor = mx / mx_input;
	printf("scaling output by %f\n", 1.0/scale_factor);
	fflush(stdout);
	output = output / scale_factor;
end

printf("writing shifted audio to %s\n", ofilename);
fflush(stdout);
%
wavwrite(output, sampleRate, bits, ofilename);

disp('ALL DONE');
end % function
%

The screenshot below shows running the chipmunk function in Octave 3.2.4 on a PC under Windows XP Service Pack 2 (Click on the screenshot image to see the full size screenshot). This screenshot shows the function called from the Octave prompt using the default values of the function’s arguments. The argument numberOverlaps controls the mathematics to compensate for the uncentered frequency components. If numberOverlaps is one, there is no compensation. The larger numberOverlaps, the more effective the compensation. The more overlaps, the more computer time and resources required by the pitch shifting. A value of numberOverlaps of thirty-two (32) was used to pitch shift President Obama’s voice in the video above.

Running the Chipmunk Function in Octave

Although easily understandable, these pitch-shifted voices sound somewhat artificial. Indeed, this artificial quality is part of the appeal of the Alvin and the Chipmunk voice.

Pitch Shifting Gets Better

Pitch shifting algorithms have improved. It is now possible to produce voices that sound much more like natural voices at the desired new pitch, very similar to the voice of Mickey Mouse. This video is President Obama speaking with a voice similar to the voice of Mickey Mouse:

This particular pitch shifting algorithm does better with producing natural sounding high pitched voices than low pitched voices.

Conclusion

There are many ways to manipulate voices using mathematics. One of the most common is pitch shifting, which has been described in detail including working source code above. Traditional pitch shifting algorithms give artificial qualities to the pitch-shifted voice. There are now new, improved algorithms that can create more natural sounding pitch-shifted voices. These voices can be used for humor, entertainment, or emphasis in movies, television, video games, video advertisements for small businesses, personal and home video, and in many other applications.

About the Author

John F. McGowan, Ph.D. solves problems using mathematics and mathematical software, including developing video compression and speech recognition technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Possibly related articles:

The Game of Life in Octave

John F. McGowan, Ph.D. — Mon, 16 May 2011 16:20:32 +0000

The Game of Life is a simple cellular automaton invented by mathematician John Conway. The Game of Life consists of a two dimensional grid of cells. Each cell can be “live” or “dead”, often represented mathematically by 1 and 0. The grid is updated periodically in discrete time steps according to a simple rule based on the cell’s neighbors, the cells immediately adjacent to the cell horizontally, vertically, and diagonally. The figure below shows a live cell (black is used for a live cell in this article) and its dead neighbors. The grid is displayed for clarity.

Cell Neighbors

The update rule for the Game of Life is:

A live cell with fewer than two live neighbors dies.
A live cell with two or three neighbors lives on to the next time step.
A live cell with more than three neighbors dies.
A dead cell with exactly three live neighbors becomes a live cell in the next time step.

The Game of Life has many interesting and entertaining properties. Amongst other features, it can implement a general purpose computer like the much more complex Pentium CPU chip. There is extensive information on the Game of Life on the Web. Interested readers are referred to this online information as well as the many traditional printed articles and books that discuss the Game of Life. This article discusses how to implement the Game of Life in Octave, a free open-source numerical programming environment that is mostly compatible with MATLAB. Octave is available in both source code and pre-compiled binaries for all three major computing platforms: MS Windows, Macintosh, and Unix/Linux. Full source code in Octave is presented and the results of several simulations of the Game of Life are shown.

Simulations of the Game of Life

There are a number of structures in the Game of Life that have interesting properties. These range from simple oscillators that cycle through a fixed set of patterns to complex self-perpetuating patterns that grow and evolve indefinitely. Several are shown below. These were simulated using an implementation of the Game of Life in Octave. The simulation generates a sequence of images showing the time evolution of the Game of Life. These image squences are either static GIF images or static JPEG images. Where possible, the images were combined into animated GIF sequences shown below using the convert utility in ImageMagick. In some cases this proved difficult and the images were converted to MPEG-4 video (MP4) using the ffmpeg command line utility.

Toad Oscillator in the Game of Life (Click on Image to See Movie)

Pulsar Oscillator (Click on Image to See Movie)

Glider in Game of Life (Click below to see movie)

Glider Movie

Gosper's Gun (Generates Gliders)

Gosper’s Gun (Click here to see movie)

Octave Code

The main Game of Life simulation function is simulate_life. This implementation uses a series of nested loops to implement the update rule in a way that will be familiar to users of procedural programming languages like the C programming language. Later, a function simulate_life_fast is presented which implements the update rule using the “matrix” operations in Octave with no loops over the rows and columns of the Octave matrix that represents the Game of Life universe.

Main Simulation

simulate_life.m

function [] = simulate_life(myseed, niter, name, ext)
% [] = simulate_life(myseed, niter [,name, ext])
% simulate Conway's Game of Life
% myseed is 2d array with values 0 or 1
% 0 is a dead cell
% 1 is a live cell
% niter is number of iterations to simulate
% name (optional) name of simulation
% ext (optionsal) extension of image file (default gif)
%
% Author: John F. McGowan, Ph.D.
% E-Mail: jmcgowan11@earthlink.net
%
% to make an animated gif of the game of life
%
% Install ImageMagick on your computer
% command prompt>convert -delay 10 -loop 0 life*.gif game.gif
% to make game.gif animated gif
%
% any good web browser (such as FireFox) can display animated GIF video
%

if (nargin < 2)
	printf("ERROR: too few arguments!\n");
	printf("Usage: simulate_life(myseed, number_of_iterations [,simulation_name])\n");
	fflush(stdout);
	return;
end

if nargin < 3
	name = 'life';
end

if nargin < 4
	ext = 'gif';
end

nx = rows(myseed);
ny = columns(myseed);

previous = myseed;
update = zeros(size(previous));

display_array = life_grid(myseed);

imshow(!display_array);
title('SEED');
pause(1);

seed_name = sprintf("%s000.%s", name, ext);

print(seed_name);

total_live = sum(sum(previous));
printf("%d live cells in seed\n", total_live);
fflush(stdout);

for iter = 1:niter

% simulate one iteration/update of game
%
for ix = 1:nx
	for iy = 1:ny
			lowx = max(ix-1,1);
			hix = min(ix+1, nx);
			lowy = max(iy-1,1);
			hiy = min(iy+1, ny);
			nlive = sum(sum(previous(lowx:hix,lowy:hiy))); % add up number of live cells in neighborhood

			% handle four (4) corner cases
			if (ix == 1 && iy == 1)	|| (ix == 1 && iy == ny) || (ix == nx && iy == ny) || (ix == nx && iy == 1)
					n11 = previous(cycle(ix-1,nx), cycle(iy-1,ny));
					n12 = previous(ix      , cycle(iy-1, ny));
					n13 = previous(cycle(ix+1,nx), cycle(iy-1,ny));
					n21 = previous(cycle(ix-1,nx), iy);
					n22 = previous(ix, iy); % center cell (current cell)
					n23 = previous(cycle(ix+1,nx), iy);
					n31 = previous(cycle(ix-1,nx), cycle(iy+1, ny));
					n32 = previous(ix      , cycle(iy+1, ny));
					n33 = previous(cycle(ix+1, nx), cycle(iy+1, ny));
					nlive = n11 + n12 + n13 + n21 + n22 + n23 + n31 + n32 + n33;
			else
			% non corner cases

				% handle cells at edge of universe (treat as closed universe)
				if ix == 1
					nlive = nlive + sum(previous(nx,lowy:hiy));
				end

				if ix == nx
					nlive = nlive + sum(previous(1,lowy:hiy));
				end

				if iy == 1
					nlive = nlive + sum(previous(lowx:hix, ny));
				end

				if iy == ny
					nlive = nlive + sum(previous(lowx:hix, 1));
				end

			end % else non corner cases

			if previous(ix,iy) == 1
				nlive = nlive - 1; % don't count center cell
				printf("live cell %d %d has %d live neighbors\n", ix, iy, nlive);
				fflush(stdout);
			end

			if nlive < 2  % cell with fewer than 2 live neighbors dies
				update(ix, iy) = 0; % cell dies
			elseif (nlive ==2 || nlive ==3) % cell lives on if it has 2 or 3 neighbors
				if(previous(ix,iy) == 0 && nlive == 3)
					update(ix, iy) = 1; % dead cell comes alive if it has exactly 3 live neighbors (reproduction)
				else
					update(ix, iy) = previous(ix,iy);
				end
			elseif nlive > 3 % cell dies due to overpopulation
				update(ix, iy) = 0;
			else
				update(ix, iy) = previous(ix,iy);
				printf("error if got here\n");
				fflush(stdout);
			end

	end % loop over columns
end % loop over rows

total_live = sum(sum(update));
printf("%d live cells at iteration %d\n", total_live, iter);
fflush(stdout);

previous = update;
filename = sprintf("%s%03d.%s", name, iter, ext); % write image sequence to disk
%imshow(!previous);
display_array = life_grid(previous);
imshow(!display_array);

title(filename);
pause(1);
print(filename);

end % loop over iterations

end % function simulate_life

The function simulate_life calls a support function cycle which uses the Octave mod function to wrap the row and column coordinates at the edges of the Game of Life universe. This closes the Game of Life universe, giving it a torus or donut shape. In the glider simulation, the glider travels off one end of the universe and reappears at the other end.

cycle.m

function [result] = cycle(n,m)
% [result] = cycle(n,m)
% 1 to n index wrap
%
result = mod(n-1,m)+1;
end

life_grid.m

function [display_array] = life_grid(previous)
% [display_array] = life_grid(previous)
%
display_array = zeros(10*size(previous));  % create a larger display array
display_array(1:10:end,:) = 1;   % create grid
display_array(:,1:10:end) = 1;
display_array(end,:) = 1;
display_array(:,end) = 1;

for i = 1:rows(display_array)-1
for j = 1:columns(display_array)-1
       idata = floor((i-1)/10)+1;
       jdata = floor((j-1)/10)+1;
       if(previous(idata, jdata) == 1)
               display_array(i,j) = 1;
       end
end
end

end % end function life_grid

Initializing Special Patterns

life_toad.m

function [result] = life_toad(myseed, toad_x, toad_y)
% [result] = life_toad(myseed, toad_x, toad_y)
% add a "toad" oscilaltor to Conway's game of life
% myseed -- the universe array (2D)
% toad_x -- x coordinate of toad
% toad_y -- y coordinate of toad
%
% Author: John F. McGowan, Ph.D.
% Web: http://www.jmcgowan.com/
% E-Mail: jmcgowan11@earthlink.net
%

result = myseed;

result(toad_x, toad_y) = 1;
result(toad_x+1, toad_y) = 1;
result(toad_x+2, toad_y) = 1;

result(toad_x + 1, toad_y + 1) = 1;
result(toad_x + 2, toad_y + 1) = 1;
result(toad_x + 3, toad_y + 1) = 1;

end % function life_toad

life_pulsar.m

function [result] = life_pulsar(myseed, pulsar_x, pulsar_y)
% [result] = life_pulsar(myseed, pulsar_x, pulsar_y)
% add a "pulsar" oscilaltor to Conway's game of life
% myseed -- the universe array (2D)
% pulsar_x -- x coordinate of pulsar
% pulsar_y -- y coordinate of pulsar
%
% Author: John F. McGowan, Ph.D.
% Web: http://www.jmcgowan.com/
% E-Mail: jmcgowan11@earthlink.net
%

pulsar = [ 0 0 1 1 1 0 0 0 1 1 1 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 0;
 1 0 0 0 0 1 0 1 0 0 0 0 1;
 1 0 0 0 0 1 0 1 0 0 0 0 1;
 1 0 0 0 0 1 0 1 0 0 0 0 1;
 0 0 1 1 1 0 0 0 1 1 1 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 1 1 1 0 0 0 1 1 1 0 0;
 1 0 0 0 0 1 0 1 0 0 0 0 1;
 1 0 0 0 0 1 0 1 0 0 0 0 1;
 1 0 0 0 0 1 0 1 0 0 0 0 1;
 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 1 1 1 0 0 0 1 1 1 0 0;
 ];

result = myseed;

result(pulsar_x:pulsar_x+rows(pulsar)-1, pulsar_y:pulsar_y+columns(pulsar)-1) = pulsar;

end % function life_pulsar

life_glider.m

function [result] = life_glider(myseed, glider_x, glider_y)
% [result] = life_glider(myseed, glider_x, glider_y)
% add a glider to Conway's game of life
% myseed -- the universe array (2D)
% glider_x -- x coordinate of glider
% glider_y -- y coordinate of glider
%
% Author: John F. McGowan, Ph.D.
% Web: http://www.jmcgowan.com/
% E-Mail: jmcgowan11@earthlink.net
%

result = myseed;

result(glider_x, glider_y) = 1;
result(glider_x+1, glider_y+1) = 1;
result(glider_x+2, glider_y+1) = 1;
result(glider_x, glider_y + 2) = 1;
result(glider_x + 1, glider_y + 2) = 1;

end % function life_glider

life_gosper.m

function [result] = life_gosper(myseed, gosper_x, gosper_y)
% [result] = life_gosper(myseed, gosper_x, gosper_y)
% add a "gosper" oscilaltor to Conway's game of life
% myseed -- the universe array (2D)
% gosper_x -- x coordinate of gosper
% gosper_y -- y coordinate of gosper
%
% Author: John F. McGowan, Ph.D.
% Web: http://www.jmcgowan.com/
% E-Mail: jmcgowan11@earthlink.net
%

gosper = [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0;
 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 1 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0;
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0;
 ];

result = myseed;

result(gosper_x:gosper_x+rows(gosper)-1, gosper_y:gosper_y+columns(gosper)-1) = gosper;

end % function life_gosper

Top Level Simulation Scripts

The simulations were generated using the following Octave scripts:

life.m

% John Conway's Game of Life in Octave
%
% Author: John F. McGowan, Ph.D.
% Web: http://www.jmcgowan.com/
% E-Mail: jmcgowan11@earthlink.net
%

nx = 20;
ny = 20;

niter = 10; % number of iterations of game to simulate

% 0 is a dead cell, 1 is a live cell
%
% blinker
% myseed = repmat(0, nx, ny); % create seed for game of life
% myseed(nx/2, ny/2) = 1;
% myseed((nx/2)+1, ny/2) = 1;
% myseed((nx/2)+2, ny/2) = 1;

% simulate_life(myseed, niter);	

% glider
myseed = repmat(0, nx, ny);

glider_x = nx/2 + 7;
glider_y = ny/2 + 7;

%myseed = life_glider(myseed, glider_x, glider_y);

% add a toad oscillator
%myseed = life_toad(myseed, nx/2, ny/2);

% blinker at corner
% myseed(1,ny) = 1;
% myseed(nx,ny) = 1;
% myseed(nx-1, ny) = 1;

% add a pulsar oscillator (period 3 iterations)
myseed = life_pulsar(myseed, 5,5);
simulate_life(myseed, 10, "pulsar");

% END OF FILE (Conway's Game of Life in Octave)

gosper_demo.m

% gosper_demo
%
% demo of Gosper Gun in Conway's Game of Life
%
% Author: John F. McGowan, Ph.D.
% E-Mail: jmcgowan11@earthlink.net
%
%

myseed = zeros(60, 60);

myseed = life_gosper(myseed, 10, 10);

simulate_life(myseed, 100, 'gosper', 'jpg');

glider_demo.m

% glider
myseed = repmat(0, 20, 20);

glider_x = 10;
glider_y = 10;

myseed = life_glider(myseed, glider_x, glider_y);
simulate_life(myseed, 100, 'glider', 'jpg');

Elegant Implementation in Octave

Gosper's Gun (Fast Implementation)

Gosper’s Gun (Click Here to See Movie)

Octave has a large number of “matrix” or “n-dimensional array” operators and functions that operate on an entire Octave matrix (two-dimensional array) or n-dimensional array without nested loops over the indices of the matrix or array. These are generally faster, more compact, and often the coding is less error prone than using nested loops. This is an implementation of the Game of Life using Octave matrix operators:

simulate_life_fast.m

function [] = simulate_life_fast(myseed, niter, name, ext)
% [] = simulate_life_fast(myseed, niter [,name ,ext])
% simulate Conway's Game of Life
% myseed is 2d array with values 0 or 1
% 0 is a dead cell
% 1 is a live cell
% niter is number of iterations to simulate
% name (optional) name of simulation
% ext (optionsal) extension of image file (default gif)
%
% Author: John F. McGowan, Ph.D.
% E-Mail: jmcgowan11@earthlink.net
%
% to make an animated gif of the game of life
%
% Install ImageMagick on your computer
% command prompt>convert -delay 10 -loop 0 life*.gif game.gif
% to make game.gif animated gif
%
% any good web browser (such as FireFox) can display animated GIF video
%

if (nargin < 2)
	printf("ERROR: too few arguments!\n");
	printf("Usage: simulate_life(myseed, number_of_iterations [,simulation_name])\n");
	fflush(stdout);
	return;
end

if nargin < 3
	name = 'life';
end

if nargin < 4
	ext = 'gif';
end

nx = rows(myseed);
ny = columns(myseed);

previous = myseed;
update = zeros(size(previous));

imshow(!myseed);
title('SEED');
pause(1);

seed_name = sprintf("%s000.%s", name, ext);

count_neighbors = [1 1 1; 1 0 1; 1 1 1];
% look up tables -- zero based index
lut_live = [ 0 0 1 1 0 0 0 0 0 0 0 0]; % live cells with less than 2 live neighbors die, 2-3 live, more than 3 die (overpopulation)
lut_dead = [ 0 0 0 1 0 0 0 0 0 0 0 0]; % any dead cell with exactly three (3) live neighbors comes alive

print(seed_name);

total_live = sum(sum(previous));

for iter = 1:niter

% simulate one iteration/update of game
%
nneighbors = conv2(previous, count_neighbors, 'same');
live_change = previous .* lut_live(nneighbors+1);  % update the live cells
dead_change = !previous .* lut_dead(nneighbors+1); % bring dead cells with 3 live neighbors to life
update = live_change + dead_change;

total_live = sum(sum(update));

previous = update;
filename = sprintf("%s%03d.%s", name, iter, ext); % write image sequence to disk
imshow(!previous);
%display_array = life_grid(previous);
%imshow(!display_array);

title(filename);
print(filename);

end % loop over iterations

end % function simulate_life_fast

Conclusion

The Game of Life is a simple, easy to implement, entertaining cellular automaton. It is easy to implement the Game of Life in Octave (or MATLAB or SciLab). External tools such as ImageMagick convert or ffmpeg can be used to easily convert the image sequences that Octave can generate into animations in commonly used formats such as animated GIF or MPEG-4 video. Even using Octave’s matrix-oriented operators to implement the Game of Life, avoidign the cumbersome and generally slow nested loops over rows and columns, the Octave implementation is still slow compared to a compiled C programming language implementation. This speed issue is probably the primary drawback to using Octave, which otherwise is very quick and convenient and has a much lower development time than low level compiled languages such as C.

About the Author

John F. McGowan, Ph.D. is a software developer, research scientist, and consultant. He works primarily in the area of complex algorithms that embody advanced mathematical and logical concepts, including speech recognition and video compression technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Admin’s message: Looking for some great mathematical reads? Check out our list of must-read math books.

Possibly related articles:

Associative Arrays and Cellular Automata in Octave

John F. McGowan, Ph.D. — Mon, 09 May 2011 17:29:18 +0000

An associative array, also known as a dictionary, map, mapping, or hash table, is a powerful data structure that is built into many modern programming languages such as Python, Perl, Ruby, and many others. An associative array is a form of content addressable memory (CAM). For example, when you see someone’s face, you often remember many other facts. This is John Smith. John Smith is your neighbor and lives at 123 Elm Street. John’s wife is Amanda who has long blond hair and so on. As associative array associates an object, often called a key, with another object, often called a value.

In many programming languages the key is a string of characters such as "John Smith" and the value is another string such as "Address:123 Elm Street;Wife:Amanda" and so forth. In some object oriented computer programming languages, the key can be any kind of object and the value can be any kind of object such as the key and the value . An associative array is largely equivalent to a single table in a relational database (RDBMS). In principle, a network of associative arrays can represent complex abstract knowledge and reasoning.

Technically, associative arrays are usually implemented using a hash table. A hash table uses modulo arithmetic to map an object such as a string of characters to the numerical index of an array of values. This array of values is not a simple one dimensional array because there can be collisions where two keys, objects such as strings of characters, “hash” to the same array index. If this happens, there are various methods such as hanging a linked list of elements off the array to handle the collision. Using a hash table means that the time to look up a value is almost constant (O(1)) regardless of the size of the hash table.

The hash table may have millions of entries, but it takes the same small number of operations to look up the associated value. First, compute the array index using modulo arithmetic on the numeric value of the “key”. Second, handle any collisions. The hash table should be large enough that collisions are rare. In principle, an associative array could be implemented in other ways, but hash tables of some kind are generally the fastest, most flexible way to implement an associative array. Hence, the terms associative array, dictionary, map, mapping, and hash table are often used interchangeably.

Octave is a free, open-source numerical programming environment that is mostly compatible with MATLAB. Octave is largely built around matrices (two dimensional arrays) and N (2,3,etc.) dimensional arrays. By default, matrices and N-D arrays in Octave are of the type double (usually a 64 bit IEEE-754 double precision floating point number).

This is due to the history of MATLAB (short for Matrix Laboratory) which started life as an interactive environment for performing two dimensional matrix algebra and computations. At first glance, Octave lacks associative arrays which is a significant deficiency for some types of programming including some types of mathematical programming. Octave does, in fact, have associative arrays. This article shows how to use associative arrays in Octave and use associative arrays to implement cellular automata, a type of discrete mathematical model.

Associative Arrays in Octave

While Octave lacks an explicitly identified associative array, dictionary, map, mapping, or hash table data type, Octave does have data structures or structs similar to structures in the C family of programming languages. For example, in Octave one can define a structure interactively like this:

octave-3.2.3:2> a.key = 'value'
a =
{
key = value
}

octave-3.2.3:3> a.key2 = 'value2'
a =
{
key = value
key2 = value2
}

The structure a now has two fields “key” and “key2″ with values ‘value’ and ‘value2′. In Octave, the data structures are implemented using a hash table and can act as a fully functional associative array. Octave provides several functions to access and manipulate structures, making it easy to use a structure in Octave as an associative array.

Built-in Function: struct ("field", value, "field", value,...)
Built-in Function: isstruct (expr)
Built-in Function: rmfield (s, f)
Function File: [k1,..., v1] = setfield (s, k1, v1,...)
Function File: [t, p] = orderfields (s1, s2)
Built-in Function: fieldnames (struct)
Built-in Function: isfield (expr, name)
Function File: [v1,...] = getfield (s, key,...)
Function File: substruct (type, subs,...)

All of these functions are useful. Most important for the purposes of this article are struct("field", value, "field", value,...) which creates a data structure explicitly, setfield(structure_name, key, value,...) which assigns a value to a key, and getfield(structure_name, key) which retrieves the value associated with key. These are used in the examples in this article to implement cellular automata.

Cellular Automata

A cellular automaton (plural, cellular automata, sometimes abbreviated as CA) is a discrete mathematical model. A typical cellular automaton consists of a grid of cells with one or more dimensions. Often, the cells have two possible values, 0 and 1, which are often displayed as black and white pixels graphically. The cellular automaton evolves over time in discrete time steps. With each time step or update, a cell changes to 0 or 1 based on the values of the cells in its neighborhood. A cellular automaton has a rule that specifies how it updates.

Cellular automata have been used in entertainment (they make pretty pictures), mathematics, physics, and a number of other fields. Probably the most well known cellular automaton is the “Game of Life”, a two dimensional cellular automaton with many entertaining and interesting properties invented by the mathematician John Conway in the 1970′s.

Stephen Wolfram, the creator of the Mathematica computer algebra system and mathematical research tool, has had a long standing interest in cellular automata. His book, A New Kind of Science, speculates that the universe might be a sort of cellular automaton and be “computationally undecidable” (in layman’s terms, math and science don’t have all the answers). Matthew Cook, who assisted Wolfram in the research for A New Kind of Science, proved that a particularly simple cellular automaton known as “rule 110″ can function as a general purpose computer just like more complex systems such as the Pentium computer chip. An implementation of the “rule 110″ cellular automaton is one of the examples in this article.

The rule for a cellular automaton can be easily represented as an associative array that maps each possible neighborhood to a new value. This is very simple and intuitive. It is easy to implement cellular automata in programming languages with built-in associative array data types. This is illustrated in Octave in the examples in this article.

Rule 30 Cellular Automaton

Rule 110 Cellular Automaton

automata.m

% Description:
%
% implementation of rule 30 and rule 110 cellular automata using an
% associative array in Octave
%
% Author: John F. McGowan, Ph.D.
% E-Mail: jmcgowan11@earthlink.net
%

universe_size = 600;
seed_start = universe_size/2;

rule30 = struct("III", "O", "IIO", "O", "IOI", "O", "IOO", "I", ...
"OII", "I", "OIO", "I", "OOI", "I", "OOO", "O"); % use Octave struct as an associative array (also knowns as dictionary or mapping or hash table)

myseed = repmat('O', 1, universe_size);
myseed(seed_start) = 'I';

myimage = simulate_ca(myseed, rule30, seed_start);
figure(1);
imshow(myimage);
title('Rule 30 Cellular Automaton');
print('rule30.jpg');

% rule 110
% proven to be Turing Complete
%

rule110 = struct("III", "O", "IIO", "I", "IOI", "I", "IOO", "O", ...
"OII", "I", "OIO", "I", "OOI", "I", "OOO", "O");

myimage = simulate_ca(myseed, rule110, seed_start);
figure(2)
imshow(myimage);
title('Rule 110 Cellular Automaton');
print('rule110.jpg');

disp('ALL DONE');

which uses the function simulate_ca defined in the following code:

simulate_ca.m

function [result] = simulate_ca(myseed, rule, niter)
% simulate_ca(myseed, rule, niter)
%
% simulate  iterations of a 1D cellular automaton specified by 
% rule starting with the seed 
%
% Author: John F. McGowan, Ph.D.
% E-Mail: jmcgowan11@earthlink.net
%
rule_name = inputname(2);

update = repmat('O', 1, length(myseed));

previous = myseed;

nx = length(myseed);
ny = niter + 1

result = zeros(ny, nx, "uint8");
lut = ones(1, 26)*255; % white background
lut(9) = 0; % map I to min gray scale level

printf("rule: %s\n", rule_name);
fflush(stdout);
printf("%s\n", previous);
fflush(stdout);

index = uint8(previous) - uint8('A') + 1;
binary = lut(index);
result(1,:) = binary;

for iter = 1:niter % iterations
for i = 1:length(myseed)-3 % loop over cells
key = previous(i:i+2);
update(i+1) = getfield(rule, key);
end % loop over cells
printf("%s\n", update);
fflush(stdout);
previous = update;
index = uint8(previous) - uint8('A') + 1;
binary = lut(index);
result(iter+1,:) = binary;
end % iterations

end % function simulate_ca

The Timing of Associative Arrays in Octave

The code below tests the timing of associative arrays in Octave. As expected if a hash table is used to implement an associative array, the lookup time is largely independent of the size of the associative array in Octave, which is good. As with many features in Octave and other mathematical scripting tools, the lookup time is quite slow.

For example, on a 3GHz Macintosh running OS X, the lookup time varied between 1 and 10 milliseconds. This is much slower than a compiled implementation of a hash table in the C programming language or another fast compiled language. Although languages such as Octave are slowly closing the speed of execution gap with compiled languages such as C, the compiled languages still win handily in some cases.

dictionary_time.m

%
% measure timing of associative arrays in Octave
% demonstrate lookup time is not slowed by number of keys (fields)
% in the associative array/dictionary. Works like a hash table.
%
% Author: John F. McGowan, Ph.D.
% E-Mail: jmcgowan11@earthlink.net
%
%

clear small_dict;
small_dict = struct('key', 'value'); % make sure a is defined

clear big_dict;
big_dict = struct('key', 'value'); % make sure a is defined

offset = uint8('A') -1;

n = 3;

if(n > 26) % 26 characters in English alphabet
n = 26;
end

[total, user, system] = cputime();
for i = 1:n
for j = 1:n
for k = 1:n
key = char([i j k] + offset);
value = ['value' key];
%     printf("key: %s value: %s\n", key, value);
small_dict = setfield(small_dict, key, value);
end
end
end
[total1, user1, system1] = cputime();

printf("CPU TIME: %f\n", total1 - total);

printf("creating big dictionary\n");
fflush(stdout);

% create big dictionary (associative array)

n = 26;

if(n > 26) % 26 characters in English alphabet
n = 26;
end

[total, user, system] = cputime();
for i = 1:n
printf("creating fields starting with %s\n", char(i + offset));
fflush(stdout);
for j = 1:n
for k = 1:n
key = char([i j k] + offset);
value = ['value' key];
%     printf("key: %s value: %s\n", key, value);
big_dict = setfield(big_dict, key, value);
end
end
end
[total1, user1, system1] = cputime();

printf("CPU TIME: %f\n", total1 - total);

% measure access time

[total, user, system] = cputime();
blatz = getfield(small_dict, 'AAA');
[total1, user1, system1] = cputime();
printf("%f seconds to retrieve %s for AAA from small dict\n", total1 -
total, blatz);

[total, user, system] = cputime();
blatz = getfield(big_dict, 'AAA');
[total1, user1, system1] = cputime();
printf("%f seconds to retrieve %s for AAA from big dict\n", total1 -
total, blatz);

disp('ALL DONE');

Conclusion

There are associative arrays, also known as dictionaries, maps, mappings, or hash tables, in Octave. This is poorly documented both in the official documentation and most online information about Octave. One can perform the same tasks and implement the same algorithms with the associative arrays (data structures or structs) in Octave that one can with the explicitly identified associative array data types in Perl, Python, Ruby, Java, and many other modern programming languages. Associative arrays are very useful for implementing certain kinds of mathematics in Octave such as, but not limited to, cellular automata.

About the Author

Admin’s message: Looking for some great mathematical reads? Check out our list of must-read math books.

Possibly related articles:

Plotting and Graphics in Octave

John F. McGowan, Ph.D. — Mon, 25 Apr 2011 19:37:10 +0000

Introduction

This is the fourth article in an occasional series of articles about Octave, a free open-source numerical programming environment that is mostly compatible with MATLAB. This series began with the article Octave: An Alternative to the High Cost of MATLAB. This article discusses plotting and graphics in Octave. Octave has extensive plotting and graphics features including two-dimensional plots, histograms, three dimensional plots, and a range of specialized plots and charts such as pie charts. This article gives an overview of the key plotting and graphics features of Octave, discusses a few gotchas, and gives several illustrative examples.

Plotting

Octave can plot functions and data using the built-in plot functio. To illustrate thiese features, this article uses data on the price of gasoline in the United States from the United States Energy Information Administration, part of the US Department of Energy. The data is taken from the Motor Gasoline Retail Prices, U.S. City Average report released March 29, 2011. This data is currently available in Adobe Acrobat PDF file format, a comma separated value (CSV) ASCII text format, and Microsoft Excel (XLS) spreadsheet format. The data from the CSV report was imported into Microsoft Excel, exported as an ASCII tab delimited text fiel and then cut and pasted into two tab-delimited ASCII files. The EIA gas price report contains several time series in a single file (leaded gas prices, unleaded gas prices, and several more). The leaded and unleaded gas prices were extracted manually in the Notepad++ text editor. These dataq files were named: leaded.txt and unleaded.txt.

Unleaded Gas Price

unleaded = dlmread('unleaded.txt');

The plot of unleaded gasonline prices above was generated using the following Octave code:

unleaded = dlmread('unleaded.txt');
unleaded_rawtime = unleaded(:,2);
unleaded_rawprice = unleaded(:,3);
unleaded_good_index = find(unleaded_rawprice < 10);
unleaded_time_x = unleaded_rawtime(unleaded_good_index);
unleaded_price = unleaded_rawprice(unleaded_good_index);
unleaded_time = fix(unleaded_time_x/100) + rem(unleaded_time_x,100)*(1./12.);
plot(unleaded_time, unleaded_price);
title('Unleaded Regular Gas Price');
xlabel('Year');
ylabel('US Dollars Per Gallon');
print('unleaded.jpg');

A few comments may be helpful. dlmread is an Octave function that reads ASCII data files. It is fairly flexible and can often automatically identify the separator used in ASCII data files such as the tab or a comma. If necessary, the user can explicitly specify the separator and other parameters of the data file. Nonetheless it is common to encounter data files with various quirks. For example, the EIA gas price report contains several time series in a single file. There are many months for which either data is not available or not reported; these are indicated by a value of 10000000 for the gas price. The code above uses the Octave find function to select the valid data. Further the year and month are combined in the format YYYYMM sso January 1970 would be “197001″, February 1970 is “197002″, and so forth. Used directly in the Octave plot function, this will produce a nonsense plot that is not useful.. Thus, the example code above uses Octave fix and mod functions to compute a time in years (month numbers are converted to fractions of a calendar year). Octave has numerous advanced functions such as find , fix, rem, and so forth that can be used to clean up and reformat data as needed.

The Octave function plot handles the actually plotting of the graph. The Octae plot function is a very versatile plotting function for two dimensional data such as time series. Both the Octave user manual and the build in help (help plot) provide detailed information on the use of plot.

Histograms

Octave has built in support for histograms. A histogram is a way of displaying the frequency of occurrence of data or events. One might, for example, be interested in how often gas prices change by one percent, two percent, or ten percent in one month.

Histogram of Gas Price Changes

By default, the Octave hist function creates a histogram with ten bins, which is often not very useful. One can specify more bins easily.

Histogram with 100 Bins

Probability density functions are normalized ito unity(1.0). With Octave one can easily generate histograms normalized to 1.0.

Normalized Histogram of Gas Price Changes

Now the values in the histogram of percent changes in gas prices are normalized to 1.0. The histogram is an estimate of the probability density function for gas price changes. One might wonder whether this distribution is the Gaussian probability denisty, also known as the Normal or Bell Curve distribution. In fact, the histogram looks narrower than a typical Gaussian and also has some outliers, a long tail, which are not typical of a true Gaussian distribution. The next two figures show the Gaussian probability density function and a histogram of synthetic data generated with Gaussian statistics using the same mean and variance as the actual gas price data.

Gaussian Probability Density Function

Histogram of Synthetic Gaussian Data

It is easy to see that the Gaussian, or Nornal or Bell Curve, distribution differs from the distribution of the gas price data. The actual data has a narrower, sharper peak and long tails. Every now and then, gas prices jump shaprly. This is a pattern seen in many financial and other kinds of assets . Many popular financial models such as the Black-Scholes option piricing model assume Gaussian or near-Gaussian distributions which generally understates the risks when a fniancial asset or commodity has long non-Gaussian tails as gasoline does.

The histograms and related plots above were generated using the following Octave code:

change = conv(unleaded_price, [-1 1]);
len = length(unleaded_price);
returns = change(2:len) ./ unleaded_price(2:len);
returns = 100.0 * returns; % convert to percent
plot(unleaded_time(2:len), returns, 'g;return;');
title('Returns on Gas');
xlabel('Year');
ylabel('Return (Percent)');
print('gas_returns.jpg');

printf('making plot 2\n');
fflush(stdout);

figure(2)
hist(returns);
title('hist(returns)');
print('hist_returns.jpg');
xlabel('Return (Percent)');
ylabel('Counts');

printf('making plot 3\n');
fflush(stdout);

figure(3)
hist(returns, 100);
title('hist(returns, 100) (100 BINS)');
xlabel('Return (Percent)');
ylabel('Counts');
print('hist100_returns.jpg');

printf('making plot 4\n');
fflush(stdout);

figure(4)
hist(returns, 100, 1.0);
title('Normalized Returns');
xlabel('Return (Percent)');
ylabel('Normalized Counts');
print('hist_norm_returns.jpg');

printf('making plot 5\n');
fflush(stdout);

figure(5)
m = mean(returns);
sigma = std(returns);
x = (-100:100);
plot(x, mygauss(x, m, sigma));
title('Gaussian with Same Mean/Standard Deviation');
xlabel('Return (Percent)');
ylabel('Probability');
print('gauss_model.jpg');
% let user know processing is done
% in case figure does not pop in front

k_returns = kurtosis(returns);

check = sigma*randn(1, length(returns)) + m;
k_check = kurtosis(check);
m_check = mean(check);
sigma_check = std(check);

printf('making plot 6\n');
fflush(stdout);

figure(6)
hist(check, 100, 1.0);
title('Histogram of Gaussian Model');
xlabel('Percent Chnage');
ylabel('Counts');
print('hist_gauss.jpg');

printf('ALL DONE');
fflush(stdout);
beep()

and the mygauss function which implements the Gaussian (Normal/Bell Curve) probability density function:

function [result] = mygauss(x, mean, sigma)

norm = 1.0/(sqrt(2*pi) * sigma);
exponent = -1.0*((x - mean) ./ sigma).^2;
result = norm * exp(exponent);

end

Eye Candy

Octave supports most common special plots such as stairs plots, bar charts, pie charts, and so forth.

Stairs Plot

This is a stairs plot of the gas price time series data for the first few years (notice the steps or stairs in the plot). THis is generated using the Octave stairs function

Stem Plot

This is the so-called stem plot (Octave stem function):

Stem Plot

Bar Chart

This is a bar chart (Octave bar function). On Windows, the bar function seems to have trouble with large amounts of data unlike the other plotting functions.:

Bar Chart

Horizontal Bar Chart

This is a horizontal bar chart (Octave barh function). On Windows, the bar function seems to have trouble with large amounts of data unlike the other plotting functions.

Horizontal Bar Chart

Full Source Code

The plots above were generated using the following Octave code:

% regular plot
figure(1)
plot(unleaded_time(1:36), unleaded_price(1:36));
title('plot(time, price) Regular Plot');
xlabel('Year');
ylabel('Price');
print('regular_plot.jpg');
%
figure(2)
stairs(unleaded_time(1:36), unleaded_price(1:36));
title('stairs(time, price) Stairs Plot');
xlabel('Year');
ylabel('Price');
print('stairs_plot.jpg');
%
figure(3)
stem(unleaded_time(1:36), unleaded_price(1:36));
title('stem(time, price) Stem Plot');
xlabel('Year');
ylabel('Price');
print('stem_plot.jpg');
%
figure(4)
bar(unleaded_time(1:10), unleaded_price(1:10));
title('bar(time, price) Bar Chart');
xlabel('Year');
ylabel('Price');
print('bar_plot.jpg');

figure(5)
barh(unleaded_price(1:10));
title('barh(price) Horizontal Bar Chart');
xlabel('Year');
ylabel('Price');
print('barh_plot.jpg');

beep();

Pie Chart

Octave can generate standard pie charts using the pie function:

Six Largest Oil Nations by Alleged Proven Reserves

This is the Octave code for the pie chart above (NOTE the use of the cellstr(‘country name’) syntax — this is needed);

% allegedly proven oil reserves pie chart
%Venezuela 	297
%Saudi Arabia 267
%Canada 	179
%Iraq 	143
%Iran 	138
%Kuwait 	104
reserves = [ 297, 267, 179, 143, 138, 104 ];
names = [ cellstr('Venezuela (297)'),
cellstr('Saudi Arabia (267)'),
cellstr('Canada (179)'),
cellstr('Iraq (153)'),
cellstr('Iran (138)'),
cellstr('Kuwait (104)') ];
pie(reserves, names, [1 0 0 0 0 0]);
title('Pie Chart of Six Larges Oil Nations');
xlabel('Billions of Barrels of Oil');
print('pie_reserves.jpg');

The third argument to pie (after names ) tells Octave to “explode” the first wedge (Venezuela).

Three Dimensional Graphics

Octave has functions to plot an display three dimensional data and functions:

Built In Sombrero Plot

Built In Peaks Plot

The two 3D plots above were generated using the built-in peaksand sombrero test functions. It is possible to compute and display almost any 3D surface using the meshgrid function and 3D display functions such as plot3, surf, mesh, contour, and quiver.

Meshgrid

For people coming from another type of programming such as the C family of langauges, the meshgrid concept and function is new and may take a little getting used to. A meshgrid is basically a very simple concept which is also very powerful. Octave represents almost everything as a “matrix” or multi-dimensional array. This is the source of much of the power of Octave. One can often avoid explicitly coding loops over the elements of the Octave matrix. This speeds development and reduces errors.

A meshgrid is a two (or higher) dimenaional array in which the elements of the array are the spatial location (x or y coordinate usually) of the associated element of a spatial grid (the mesh grid). Here is some Octave code which explicitly computes and plots the sombrero function:

ticks = [-10.0:0.5:10.0];
[x, y] = meshgrid(ticks, ticks);
z = sin (sqrt (x.^2 + y.^2)) ./ (sqrt (x.^2 + y.^2));
surf(x,y,z);
title('sombrero using meshgrid');
print('sombrero_meshgrid.jpg');

In this example, the Octave function meshgrid returns two arrays x and y which contin the x and y corrdinates respectively for the mesh grid elements. In this case, the x and y positions of the grid are specified by the one dimensional ticks array. The ticks run from -10 to 10.0 in steps of 0.5., a total of 41 ticks. The x array generated by meshgrid is a 41 by 51 array with the x coordinate of each element. The y array is a 41 by 41 element array with the y coordinate of each element.

The meshgrid enables one to express 3D surfaces or functions in Octave in a simple intuitive compact way:

z = sin (sqrt (x.^2 + y.^2)) ./ (sqrt (x.^2 + y.^2));

In this example, z is a two dimensional array (matrix) with the function value at the xand y coordinates specified by each grid point in the arrays xand y. Then, one can display the surface using display functions such as surf, mesh, and so forth. It usually takes some practice to get used to using meshgrid if one is not familiar with the concept.

Sombrero Meshgrid Using Surf

Sombreror Meshgrid Using Mesh

Sombrero Using Plot3

Contour Plot

Octave can create contour plots using the contourfunction.

Sombreror Meshgrid Using Contour

Vector Field Plot

Octave can create vector field plots using the quiver (as in quiver of arrows) function:

Sombrero Vector Field Using Quiver

These 3D plots were generated with the following Octave code:

% 3d graphics

% tests
figure(1);
sombrero();
print('sombrero.jpg');

pause(1);

figure(2);
peaks();
print('peaks.jpg');

% meshgrid
figure(3);
ticks = [-10.0:0.5:10.0];
[x, y] = meshgrid(ticks, ticks);
z = sin (sqrt (x.^2 + y.^2)) ./ (sqrt (x.^2 + y.^2));
surf(x,y,z);
title('sombrero using meshgrid');
print('sombrero_meshgrid.jpg');

% plot3
figure(4);
ticks = [-10.0:0.5:10.0];
[x, y] = meshgrid(ticks, ticks);
z = sin (sqrt (x.^2 + y.^2)) ./ (sqrt (x.^2 + y.^2));
plot3(x,y,z);
title('sombrero using plot3');
print('sombrero_plot3.jpg');

% mesh
figure(5);
ticks = [-10.0:0.5:10.0];
[x, y] = meshgrid(ticks, ticks);
z = sin (sqrt (x.^2 + y.^2)) ./ (sqrt (x.^2 + y.^2));
mesh(x,y,z);
title('sombrero using mesh');
print('sombrero_mesh.jpg');

% contour
figure(6);
ticks = [-10.0:0.5:10.0];
[x, y] = meshgrid(ticks, ticks);
z = sin (sqrt (x.^2 + y.^2)) ./ (sqrt (x.^2 + y.^2));
contour(x,y,z);
title('sombrero using contour');
print('sombrero_contour.jpg');

% quiver
figure(7);
ticks = [-10.0:0.5:10.0];
[x, y] = meshgrid(ticks, ticks);
z = sin (sqrt (x.^2 + y.^2)) ./ (sqrt (x.^2 + y.^2));
theta = atan2(y, x);
quiver(x,y, z.*cos(theta), z.*sin(theta)); 

title('sombrero vector field using quiver');
print('sombrero_quiver.jpg');

% all done
disp('ALL DONE!');
beep();

Debugging Octave

Octave has cryptic error messages. These messages almost always correctly identify the line of code that is in error. The verbal descripton of the error is often incoprehensible and may be wrong. The column number reported for the location of an error in the line is often wrong, for example indicating the start of the expression on the right hand side of an assignment statement where the problem is later in the line of code.

If a user cannot spot the error by reading the line of code, a common occurence, it is usually best to convert the line of code into several lines of code with each lnew line of code representing a sub-expression o f the original line of code. This approach will usually narrow the error/bug down to a specific symbol and identify the specific error.

Octave supports both true-matrix operations and element by element (aka element-wise) operations. For example, A*B is true-matrix multiplication if A and B are matrices. A.*B is element by element multiplication in which each element is multiplied by its corresponding element in the other matrix. It is easy to mistakenly use * where one should use .* or .* where one should use * in Octave. Pay close attention to th edistincition between the true matrix and element-by-element operators.

Conclusion

Octave has extensive built in plotting and graphics functions. There are a few weaknesses, notably some problems with the bar chart functions, at least in the Windows version of Octave 3.2.4. Users coming from a different type of programming background such as the C family of languages may need a little time and practice to adjust to the meshgrid concept. The plotting and graphics funsions of Octave are more than adequate for all common scientfic, engineeering, and general analytical tasks, both two and three dimensional.

About the Author

Sponsor’s message: Check out Math Better Explained, an insightful ebook and screencast series that will help you see math in a new light and experience more of those awesome “aha!” moments when ideas suddenly click.

Possibly related articles:

The Scope of Mathematical Programming Projects

John F. McGowan, Ph.D. — Wed, 09 Feb 2011 18:00:21 +0000

This article discusses the scope of mathematical programming projects using the example of several successful open source/free software projects. In the author’s experience, it is common to encounter rather optimistic expectations of the cost, schedule, and risk level of mathematical research and development and programming projects; the two categories, mathematical research and development and mathematical programming, are heavily blurred together today, especially for practical and applied projects. This article provides a rough picture of the scope of mathematical programming projects based on actual historical data rather than popular culture, anecdote, or folk wisdom.

There are strong practical reasons for applied mathematical research and development and programming projects. Many potentially beneficial projects exist. These projects often suffer from the “cure for cancer” problem. With several hundred thousand people each year in the United States alone succumbing to cancer, there is little question that there is a large market for a cure for cancer. The problem is that we do not know how to cure cancer. Similarly, successful mathematical research and development and programming projects offer everything from profitable investment advice to speech recognition for mobile devices and household appliances to working fusion reactors and other new energy sources. Indeed, a cure for cancer is something that mathematical methods may offer in the future through molecular modeling or other quantitative approaches. Given the huge potential markets for successful mathematical projects, it is common to encounter individuals, organizations, and companies with great interest in particular, usually practical mathematical projects. These projects are highly unlikely to succeed without accurate ideas about the scope of the projects.

The Scope of Some Successful Free Open-Source Mathematical Software Projects

Program	Lines of Code	Core Lines of Code	Calendar Duration	Number of Contributors
FFMPEG 0.6.1 Video Encoder	373,742	368,457	at least 2004-2011	50
x264 h.264 Video Encoder (x264-snapshot-20110204-2245)	67,986	62,968	at least 2004-2011	18
Independent JPEG Group JPEG encoder/decoder v8c	61,102	52,304	at least 2000-2011	13
Open CV 2.2.0 Computer Vision Library	884,808	396,399	at least 1999-2011	80
Insight Toolkit 3.20.0 Image Segmentation and Registration Toolkit	698,143	685,466	at least 1999-2011	at least 14
Pythia/Lund Monte Carlo 8.145 Particle Physics Event Simulation (C++ version)	141,353	46,258	1977-2011	5
Pythia/Lund Monte Carlo 6.327 Particle Physics Event Simulation (last FORTRAN Version)	60,455	60,455	1977-1996	5
EGS (Electron Gamma Shower)	38,921	31,151	1950′s-2011	unknown
LAPACK 3.3.0 Linear Algebra Library	459,993	458,645	at least 1970′s to present	many contributors (probably over 100)
AESCRYPT Encryption/Decryption Utility	4,331	4,286	2001-2009	at least 2
GNU Privacy Guard (GNUpg) v. 1.4.11	148,374	120,441	at least 1998 to 2008	47
Octave 3.2.4 Numerical Programming Tool	539,233	453,160	at least 1980s to present	many contributors (probably over 100)

Notes

The free, open-source CLOC (Count Lines of Code) utility was used to count the number of lines of code in each project. CLOC lists the number of lines of code in each programming language in the project such as C, C++, Bourne Shell, HTML, and so forth. CLOC does not count blank lines or comment lines. Some projects include sizable amounts of installation code (in the Unix Bourne Shell for example), HTML documentation, and so forth which is counted in the total number of lines of code reported by CLOC. The actual mathematical code is typically implemented in a few languages such as C, C++, FORTRAN, or MATLAB. The term “Core Lines of Code” refers to the lines of code in these languages, as reported by CLOC, which is presumed to contain the actual mathematical software.

In general, open source projects provide a wealth of detailed information that is difficult or impossible to acquire for many commercial proprietary projects. In particular, one can see the source code, count the lines of code or other measures of size and scope, and often read comments, change logs, logs of version control systems, and so forth. Nearly all open source projects give a list of contributors somewhere in the documentation and provide rough information on the calendar duration of the project. There is usually precise information on releases and release dates. Unfortunately, it is difficult to get a reasonably exact measure of the actual effort expended on the project. Most open source projects do not publish information on exact hours worked, dollars expended, even if records exist. Several of the examples were fully or partially funded either by government funding agencies (e.g. the National Library of Medicine for the Insight Toolkit) or private sources (e.g. Intel for OpenCV), so such detailed information may be available in some cases.

The Examples

The examples were chosen as successful free open-source projects widely used within their field or application with a quality comparable to or superior to good commercial software products. Several such as FFMPEG and x264 are highly applied and used in the everyday world. Several such as the Pythia/Lund Monte Carlo are primarily scientific research tools. Some such as Octave and LAPACK span both worlds.

FFMPEG is a widely used open source audio/video encoding utility and collection of libraries. FFMPEG can encode and decode a wide range of different audio and video formats and compression schemes including h.264. It incorporates a number of other utilities and libraries. x264 is a widely used open source h.264 video encoder. The Independent JPEG Group disributes a widely used open source JPEG image encoder and decoder. Open CV is a widely used computer vision library incorporating many of the current state of the art computer vision algorithms; it is used in research and in a few commercial products. The Insight Toolkit is a toolkit of image segmentation and registration algorithms, somewhat similar to Open CV in practice, geared towward medical imaging.

The Pythia/Lund Monte Carlo is a widely used program for simulating the formation of jets of subatomic particles and other processes in experimental and theoretical particle physics, for example at the Large Hadron Collider (LHC) at CERN. Two versions, the original FORTRAN version and the more recent rewrite in C++, are listed. Electron Gamma Shower or EGS is a widely used program for simulating the interactions of electrons and photons (gamma rays and x-rays) with matter. It was originally developed for nuclear and particle phyics at the Stanford Linear Accelerator Center (SLAC), but is now widely used for medical radiation studies. LAPACK is a widely used FORTRAN library of linear algebra and other basic numerical algorithms; it is often found in other programs as well. AESCRYPT is a free open-source implementation of the Advanced Encryption Standard (AES) for data encryption. GNU Privacy Guard (GNUpg) is a free, open-source implementation of the OpenPGP encryption standard. Octave is a free, open-source numerical programming tool that is mostly compatibly with MATLAB. Octave has been discussed in previous articles by this author starting with Octave: An Alternative to the High Cost of MATLAB.

Actual Effort Estimation with Basic COCOMO

The Constructive Cost Model (COCOMO) is a software cost estimation model developed by Barry Boehm. Basic COCOMO is the original, very simple cost estimation model published by Boehm in his 1981 book Software Engineering Economics. It gives a simple, crude estimate of the effort in man-months as a function of the number of lines of code in a project. The following table gives the estimated effort in man-months/man-years from applying the “organic” Basic COCOMO model to the number of lines of code in each mathematical open source project in this article:

Program	Basic COCOMO Man-Months	Basic COCOMO Man-Years
FFMPEG 0.6.1	1,204	100
x264	201.5	16.75
IJG v8c	179.8	15
Open CV 2.2.0	2,982	248.5
Insight Toolkit	2,324	193.7
Pythia/Lund 8.145	443	36
Pythia/Lund 6.327	178	14.8
EGS	112	9.3
LAPACK 3.3.0	1,637	136.4
AESCRYPT	11	0.9
GNU Privacy Guard (GNUpg) 1.4.11	456	38
Octave 3.2.4	1,771	147.6

The following Octave/MATLAB function was used to compute the estimated man-months using the Basic COCOMO “organic” model:

function [man_months, dev_time, people_required] = cocomo(kloc, type)
    % [man_months, dev_time, people_required] = cocomo(kloc [, type])
    %
    % kloc (thousands of lines of code)
    % type (type of project: organic, semi-detached, embedded)
    %

    if nargin < 2
    	type = 'organic'
    end

    c = 2.5;

    if strcmp(type, 'organic')
    	a = 2.4;
    	b = 1.05;
    	d = 0.38;
    end

    if strcmp(type, 'semi') % semi detached
    	a = 3.0;
    	b = 1.12;
    	d = 0.35;
    end

    if strcmp(type, 'embedded')
    	a = 3.6;
    	b = 1.2;
    	d = 0.32;
    end

    man_months = a*(kloc)^b;
    dev_time = c*(man_months)^d;
    people_required = man_months / dev_time;

end

Conclusion

While this data sample is clearly limited and a larger study is desirable, it should nonetheless be evident that successful mathematical programming projects are usually substantial. Even the smallest project on the list, the AESCRYPT encryption utility, probably took several man-months to fully develop; Basic COCOMO would estimate almost one year. Thus, expectations of a few weeks are generally unrealistic. Indeed, expectations of three calendar months, a fiscal quarter, the current fetish of American business, are usually unrealistic. On the other hand, expectations ranging from six months to several years may be realistic depending on the specific project.

In part because of heavy government funding of mathematical research and development, there are a large number of open-source, free mathematical programming projects available. This provides an excellent database of information on the size and scope of such projects, something often difficult to find for business applications where most products and projects are proprietary. Anyone considering such a mathematical project is well advised to examine comparable open source projects if they exist to determine the size and scope to the extent possible. Unfortunately, open source projects often can give only a rough measure of the actual effort (mythical man-months) used in the project. The Basic COCOMO model can provide a very rough way of estimating the actual effort of the open source project from the lines of code, but clearly a more direct way of measuring the actual effort is needed.

About the Author

Possibly related articles:

Using Maxima Output in Octave

John F. McGowan, Ph.D. — Mon, 31 Jan 2011 18:36:58 +0000

Introduction

This is the third in a series of articles on Octave starting with Octave, An Alternative to the High Cost of MATLAB. Octave is a free, both free as in beer and free as in speech, MATLAB compatible numerical programming tool available under the GNU General Public License. In part because MATLAB has become the de facto industry standard for numerical programming, Octave is of particular interest to individuals, companies, and organizations engaged in numerical and mathematical programming and research and development.

Octave has some limitations. The base Octave tool has no symbolic manipulation features. It is not a computer algebra system (CAS) such as Mathematica or Maple. It cannot, for example, perform symbolic integration, symbolic differentiation, factor polynomials, and so forth. Octave does have the symbolic toolbox available through the Octave Forge repository of Octave toolboxes, but the symbolic toolbox is quite limited. A better option for symbolic manipulation tasks is to use the Maxima computer algebra system in combination with Octave. Octave also lacks the ability to generate TeX or LaTex mathematical output, the de facto standard of mathematical publication. Maxima can also generate TeX output for inclusion in papers or WordPress blog posts.

Maxima

Maxima is a computer algebra system descended from MACSYMA, one of the original computer algebra systems. MACSYMA was developed at MIT in part for use in theoretical physics. Maxima is available both as source code and pre-compiled binaries for all three major computer platforms: Unix/Linux, Microsoft Windows, and Mac OS. Maxima is free software, both free as in beer and free as in speech, available under the GNU General Public License (GPL). wxMaxima is a Graphical User Interface (GUI) for Maxima available as both source code and pre-compiled binaries for all three major computer platforms. wxMaxima has human readable menu items and buttons for many common symbolic manipulation and mathematical functions. wxMaxima also has “notebooks” similar to Mathematica notebooks. There is considerable documentation on Maxima; interested readers are referred to the excellent online and published documentation on Maxima. This article is focused on using Maxima as an adjunct to Octave.

wxMaxima on a Macintosh

Maxima can perform both symbolic differentiation and integration. Symbolic differentiation is illustrated in the screen shot of Maxima above. Some optimization algorithms, used, for example, for model fitting, require the derivative of the function being optimized; the function is usually being minimized. If the function is rather complex, deriving the derivatives of the function with respect to the parameters over which the optimization is performed by hand can be time consuming, tedious, and error prone. The author has used Maxima successfully to perform the differentiation of a model function. One can then convert the Maxima output, the derivative produced by Maxima’s symbolic differentiation, into an expression that can be used in Octave by using the fortran(expression) command in Maxima. The Maxima fortran command generates FORTRAN code for the Maxima expression. In many cases, the FORTRAN expressions are identical to Octave/MATLAB mathematical expressions. In some cases, the FORTRAN code generated by Maxima must be edited slightly to create valid Octave/MATLAB code.

For example, the Cauchy-Lorentz distribution is a commonly used mathematical model of a peak.

The Cauchy-Lorentz distribution is the frequency response of a forced-damped harmonic oscillator. It is widely used in physics, mathematics, and engineering under a number of different names.

Cauchy Lorentz Distribution

In using the Cauchy Lorentz distribution to model a peak in data, one typically wants to determine the values of the parameters A, mu, and W representing the magnitude of the peak (A), the position of the peak (mu), and the width of the peak (W) that best fits the data. To do this, some model fitting algorithms need the derivatives of the Cauchy Lorentz distribution with respect to each parameter A, mu, and W.

Differentiation of Cauchy Lorentz in Maxima

This is the derivative of the Cauchy Lorentz distribution with respect to the width parameter W from symbolic differentiation in Maxima:

This derivative is moderately complex. Calculating this derivative by hand is time consuming and error prone. Imagine computing the derivative of an extremely complex mathematical model with hundreds of terms by hand. The probability of error even by a highly-skilled mathematician is very high. It was for this reason that tools like Maxima were developed.

This is the FORTAN code generated by applying the Maxima fortran(expression) function to the Maxima expression for the derivative of the Cauchy Lorentz with respect to the width W

2*(x-mu)**2*A/(((x-mu)**2/W**2+1)**2*W**3)

This is actually valid Octave/MATLAB code. If the variables x, mu, A, and W are scalar variables in Octave, this FORTRAN expression will evaluate correctly. Here is the calculation in Octave when x, mu, A, and W are all scalar variables with the value 1.0.

octave-3.2.4.exe:25> x = 1
x = 1
octave-3.2.4.exe:26> 2*(x-mu)**2*A/(((x-mu)**2/W**2+1)**2*W**3)
ans = 0
octave-3.2.4.exe:27>

However, in Octave the variable x is often a vector. If x is a vector, the expression above will produce an error in Octave:

octave-3.2.4.exe:25> 2*(x-mu)**2*A/(((x-mu)**2/W**2+1)**2*W**3)
error: for A^b, A must be square
octave-3.2.4.exe:25>

The reason for this error is that in Octave and MATLAB some of the operators such as * and / are not by default interpreted as element by element operators when applied to vectors and matrices. For example, the operator * is matrix multiplication by default in Octave and MATLAB. An element by element operator is an operator that is applied separately to each element in each vector or matrix that is an operand. In Octave and MATLAB, the element by element operators are .*, ./, .+, .-, and so forth. For example, if one has two vectors a and b in Octave, the operator * will give an error:

octave-3.2.4.exe:34> a
a =

1  2  3  4  5

octave-3.2.4.exe:35> b
b =

1  2  3  4  5

octave-3.2.4.exe:36> a * b
error: operator *: nonconformant arguments (op1 is 1x5, op2 is 1x5)
octave-3.2.4.exe:36>

However, in Octave and MATLAB, one can multiply each element of each vector by the corresponding element of the other vector using the element by element operator .* thus:

octave-3.2.4.exe:36> a .* b
ans =

1   4   9  16  25

octave-3.2.4.exe:37>

In the output of the element by element (or elementwise) operator .*, the first element is 1*1, the second element is 2*2, and so forth.

Thus, the FORTRAN expressions generated by Maxima are not valid Octave/MATLAB code for vectors and matrices, only for scalar variables. One can convert the FORTRAN expression to a valid Octave expression for vectors by converting the non-elementwise operators to element by element operators where the operands are vectors, usually by adding a preceding dot. Here is the edited code for the example derivative:

function [result] = mydiff_edited(x, A, mu, W)
% function [result] = mydiff(x, A, mu, W)
% FORTRAN code from wxMaxima edited to support an array x as input
%

result = 2*(x-mu).**2*A ./ (((x-mu).**2 ./ W.**2+1).**2*W**3)

end

If x is defined as a vector in Octave such as:

x = [0.0:0.1:10.0]; % x values from 0.0 to 10.0 in steps of 0.1

and then this function is used to compute the value of the derivative of the Cauchy Lorentz distribution with respect to the width parameter W:

data = mydiff_edited(x, 1.0, 1.0, 1.0);
plot(data);

Plot of Vector Derivative

Maxima can also generate valid TeX code for mathematical publication through its built in tex(expression) command. This command can also be invoked through a menu item in the wxMaxima GUI (shown below).

wxMaxima with TeX Menu Item Displayed

Some TeX generated by Maxima:

tex(1/(1+x^2));

generates the TeX code:

$${{1}\over{x^2+1}}$$

which displays in WordPress after removing the $$ tags which WordPress does not need as:

All of the mathematical formulas in this article are TeX generated by Maxima in this way.

Conclusion

Octave has extensive numerical analysis and programming features. Octave has the special advantage that it is mostly compatible with MATLAB, which is currently the de facto industry standard for numerical programming. Most Octave scripts will run under MATLAB and many MATLAB scripts will run under Octave with no changes. If a user or software developer has an occasional need for symbolic manipulation features such as symbolic integration and differentiation, one can use Maxima as an adjunct to Octave. Similarly, one can use Maxima to generate TeX code for mathematical publications. If one needs to perform extensive symbolic manipulation, one may need to use Maxima or similar tools as one’s primary tool.

Both Octave and Maxima have the advantage that they are free, both free as in beer and as in speech, and available as source code. There are many cases where a merger, change in corporate strategy, bankruptcy, or even the whim of an executive has resulted in a proprietary development platform being discarded or deprecated to the detriment of end users, developers, and other customers. A well known example is FoxPro, once one of the leading database programs, which Microsoft acquired and has now announced will be discontinued in favor of Microsoft’s other database products. In contrast, open source development tools such as Octave and Maxima can be kept alive and indeed improved by their end users, developers, and customers if needed.

About the Author

Possibly related articles:

Using Octave, a Free MATLAB Alternative

John F. McGowan, Ph.D. — Mon, 24 Jan 2011 17:09:14 +0000

Introduction

Octave is a free, both free as in beer and free as in speech, MATLAB compatible numerical programming tool available under the GNU General Public License. MATLAB, an abbreviation for “Matrix Laboratory”, is currently the most widely used commercial, proprietary numerical programming tool. Even a single license for the MATLAB software is a substantial cost. MATLAB is essentially a scripting language similar to Perl, Python, or PHP with a comprehensive, highly integrated set of numerical, statistical, mathematical, and graphical functions including, for example, the Fourier transform, matrix inversion, and histograms. In part because MATLAB has become the de facto industry standard for numerical programming, Octave is of particular interest to individuals, companies, and organizations engaged in numerical and mathematical programming and research and development.

At this time (2011), there is considerable interest in using machine learning and other advanced mathematical techniques to increase sales by detecting and predicting the buying patterns of end users of mobile devices such as iPhones and Android phones, social networks such as Facebook and LinkedIn, and search engines such as Google and Yahoo. Octave has long been used for data analysis, modeling, and machine learning. In addition to its extensive built-in features, Octave has add on packages for optimization and mathematical model fitting, econometrics, signal processing, image processing, neural networks, and a number of other areas used in the “mobile social search” space. Octave is also well suited for the development and implementation of algorithms for image and video compression, audio compression, image and video processing, audio processing, speech recognition, pattern recognition, artificial intelligence, design and simulation of machines, and many other practical uses.

In general, it is faster and easier to develop mathematical algorithms in Octave and similar tools than compiled languages such as C, C++, and Fortran. Because research and development usually involves large amounts of trial and error, Octave and similar tools have been used traditionally to prototype algorithms, analyze data from experiments, and so forth, and only occasionally for “production code” or commercial products. Traditionally, once a proof of concept existed in Octave or a similar tool, it was often translated to a faster compiled language such as C, C++, or Fortran, often a time consuming, tedious, and costly process. Octave today is quite fast and modern computers are extremely fast and inexpensive, so that there are many projects, such as data mining in the backend of a web site, that can probably skip the time consuming and costly step of conversion to a compiled language. In the future, just in time (JIT) compilers and similar capabilities will probably be added to Octave and similar tools so that the remaining speed difference between Octave and traditional compiled languages such as C, C++, and Fortran will vanish.

A significant problem with using Octave and many other free programs is making Octave work cost-effectively with other programs. It is rarely possible to use Octave for real-world projects as a purely standalone program. It is often necessary to share data and transfer the control flow between Octave and other programs including operating systems, databases, spreadsheets, web sites, and so forth. Many programs, especially proprietary programs, achieve customer lock-in through custom, sometimes secret, data formats and so forth. Microsoft Office is notorious for introducing new Office file formats with each “upgrade” that are not backward compatible with earlier versions of Office. Thus, users find themselves receiving spreadsheets, documents, and so forth that they cannot read with their old version of Office, forcing an “upgrade” at substantial cost.

This proliferation of ever-changing data formats, API’s, programming languages, and miscellaneous other standards, presents a continual problem for using free software. End users often spend hours, even days or weeks, figuring out how to convert from one format to another. Software developers often spend hours, days, weeks, and even months writing functions to read and write unsupported formats, whether new or old. This makes it difficult for end users or software developers to add real value by inventing substantive new features, capabilities, and algorithms.

This article covers a number of methods, specific commands and specific programs to make Octave work cost-effectively with other programs. There are many excellent on-line and published documents on the syntax of Octave and its add-on packages. Readers with specific questions of this type are referred to the excellent existing documentation. Rather, this article discusses some of the issues with making Octave work with other programs that are often not covered explicitly or well in the standard Octave or free software literature.

Using Simple Human Readable Files to Reduce Costs

There are a number of advantages to using simple human readable files (typically ASCII or UTF8) such as standard tab delimited files or comma separated values (CSV) files to export data, import data, and store data. One can view the files in any text editor or word processor. Most programs such as spreadsheets, databases, and many others can easily read and write tab-delimited and comma separated values (CSV) files. Many scripting languages have direct support for tab-delimited and sometimes comma separated values files. Essentially all computer programming languages can easily read and write tab-delimited files; it is just a few lines of code. Essentially all computer programming languages can read and write comma separated values files; it requires more programming, more lines of code, but it can be done. Octave can read and write tab delimited files using its built-in dlmread and dlmwrite functions. Octave can also read and write comma separated values files using its built-in csvread and csvwrite functions.

Historically, disk drives were small. Networks had very limited bandwidth. Computers had limited speed to read and process files. Hence, binary file formats had very clear advantages over bulkier human readable formats such as tab delimited files. With terabyte drives, 100 Megabit per second networks, and inexpensive 3 GHz multi-core computers, this is much less true today. Thus, there is a strong case for using simple human readable file formats such as tab delimited files to reduce high end user and development costs, avoid customer lock-in (a costly proposition if you are the customer), and limit the costly perpetual upgrade cycle to genuine improvements in the products and services.

Simple human readable files, notably tab delimited files, enable users and developers to share data between Octave and other programs quickly and easily. There are a few caveats to be aware of.

First, the basic data type in Octave is a “matrix,” originally a double precision two-dimensional array. The MATLAB/Octave “matrix” has been extended to multiple dimensions and also to support characters and strings. The character/string support in MATLAB and Octave is something of a patch. Thus, the dlmread and dlmwrite commands in Octave handle matrices or arrays of numbers fine but map character strings in tab-delimited files to zero (0). This is often not an issue for the numerical programming and analysis that Octave is usually used for. The string and text manipulation capabilities in Octave are limited, so one should consider other tools such as Perl, Python, or Ruby for heavily text or string oriented work.

The precision of floating point numbers can be an issue using human readable files such as tab delimited files that represent floating point numbers as character strings such as “1.123456″ or “1.1234567890123456″. Octave, MATLAB, and most computer programs use the IEEE-754 standard for floating point numbers, or extensions of this standard. In this standard, a single precision floating point number (e.g. a float in the C family of programming languages) requires six decimal digits after the decimal point to represent the floating point number as a character string such as “1.123456″. An IEEE-754 double precision floating point number requires 16 decimal digits after the decimal point to represent the double as a character string such “1.1234567890123456″.

Many programs and programming languages default to writing floating point numbers as character strings with six decimal digits such as “1.123456″. For example, the standard printf() function in the C programming language defaults to six decimal digits of precisoin when its %f format specifier for floating point numbers is used:

double value = 1.12345678901234456;
print("my number: %f\n", value); // defaults to six decimal digits of precision

usually prints:

my number: 1.123456

Thus, if a mathematical algorithm in Octave depends on double precision, it may be necessary to prepare a human readable input data file with 16 decimal digit character strings for the floating point numbers.

In Octave, one can use the command format long to display numbers at full precision:

Octave>format long

Exchanging Binary Files Between Octave and Other Programs

Octave supports a relatively small set of binary file formats directly. This is a common problem with free software. Both users and software developers can find themselves spending hours, days, even weeks on occasion (time is money) converting to and from other binary formats used by other programs. There are however a number of free programs that can quickly convert between a wide range of binary formats, thus greatly reducing or eliminating this overhead.

The free GIMP (The GNU Image Manipulation Program) program can read and write nearly all widely used still image file formats and many obscure still image formats. GIMP is available as both pre-compiled binaries and source code for all three major computing platforms: Microsoft Windows, Linux/Unix, and Mac OS X.

The widely used, free Audacity audio-editing program can read and write raw audio files, Microsoft uncompressed WAV files, and the open Ogg Vorbis audio file format. By default Audacity can read MP3 audio files but it cannot write them due to the onerous MP3 licensing restrictions. Users can install the LAME MP3 encoding library on all three major computing platforms — Microsoft Windows, Linux/Unix, and Mac OS X — and then Audacity can use this add on library to encode audio as MP3. Audacity is available as source code and precompiled binaries for all three major computing platforms.

The widely used, free sox (Sound Exchange) audio conversion utility can read and write a broad range of audio formats. Like Audacity, it needs the LAME encoding library to be separately installed to encode MP3 files due to the MP3 licensing restrictions. Users should consider using the free ogg vorbis audio format to avoid the MP3 licensing issues altogether. sox is available as both source code and precompiled binaries for all three major computing platforms.

The widely used, (mostly) free ffmpeg video and image sequence conversion utility can read and write a broad range of video file formats. Like Audacity and sox, certain video encoding formats have licensing restrictions. Again, users may wish to consider the free ogg theora video format to avoid some of the licensing issues with better known video formats such as h.264.

Installing Octave Add-On Packages

Octave has extensive built-in capabilities. It also has a large number of add-on packages available through the Octave Forge project and web site. These include an extensive library of standard optimization functions such as the Nelder-Mead method used in model fitting, statistics, signal processing, econometrics, and so forth. Some of the Octave Forge packages are rather limited, but a number are very extensive and a number extend Octave’s compatibility with MATLAB.

For example, MATLAB has a function xcorr which computes the cross-correlation of a data series or data set. Octave does not have xcorr in its base set of built-in functions. There are a number of implementations of xcorr on the web as xcorr.m files, but xcorr is also found in the Octave Forge signal package which provides a large number of standard signal processing functions.

The author found the explanations of the Octave pkg install procedure on the Octave and Octave Forge web sites a bit confusing. Here is a fuller explanation. Octave packages are downloaded as standard *.tar.gz gzipped (compressed) Unix tar (tape archive) files. The Unix tar (tape archive) command combines a group of files and/or folders and their contents into a single huge file without compressing the files. They are loosely concatenated together. Historically, the tar command was used to store files, folders, even entire file systems on backup tapes. The Unix gzip utility compresses files. On Unix, a *.tar.gz or *.tgz file is a file that has been tarred and then compressed using gzip. The Octave packages are distributed as *.tar.gz files.

Octave has a built in command pkg install which installs the packages. NOTE: pkg install works directly on *.tar.gz or *.tar files. There is no need to uncompress or unpack the *.tar or *tar.gz files. On the Macintosh, the Mac OS may automatically unzip the file when downloaded, creating a *.tar file.

In Octave, simply switch the folder where the *.tar.gz or *.tar file has been downloaded, then type

Octave> pkg install package-name.tar.gz

Octave> pkg install pakcage-name.tar

if the operating system has automatically unzipped the package.

The Octave pkg install command will automatically create the folders for the new package in the Octave installation and copy the package files to these folders and perform all other setup.

Many Octave packages contain C, C++, or Fortran source code. Octave automatically installs a compiler when it is installed. It can compile these files. There is no need for the Octave user to compile them by hand, run the make utility or anything like that. Octave has a built-in mkoctfile command to compile C, C++, or Fortran extensions to Octave and add them to the built-in commands of Octave. The Octave pkg install command will run mkoctfile if needed to build and install the package. On the Macintosh (see below) it may be necessary to modify the mkoctfile command to install many of the Octave packages.

Extending Octave with Fast, Compiled Languages

Developers may want to extend Octave by adding C, C++, or Fortran programs to the built-in functions of Octave. This may be done because the compiled programs will be faster to execute than Octave or to add existing C, C++, or Fortran code to Octave.

mkoctfile [-options] file

The mkoctfile function compiles source code written in C, C++, or Fortran. Depending on the options used with mkoctfile, the compiled code can be called within Octave or can be used as a stand-alone application. mkoctfile can be called from the shell prompt or from the Octave prompt.

Making Octave Work with Emacs

Emacs is a widely used free source-code and text editor. Emacs is available on all three major computing platforms: Microsoft Windows, Unix/Linux, and Mac OS. The author has had good experiences using the free Aquamacs version of Emacs on the Macintosh. Emacs has an “everything including the kitchen-sink” philosophy. Emacs has extensive built-in capabilities. There is extensive and excellent on-line and published documentation on Emacs. Again, this article will only discuss some key gotchas that are likely to arise using Emacs in combination with Octave.

Emacs has a MATLAB mode for editing MATLAB or Octave source code. The MATLAB code editing mode handles indenting, highlights keywords, and so forth. Emacs also has an Objective C mode. Since Objective C “method” or “message” source files also use the .m file extension used by MATLAB and Octave, this can cause problems. There is also an Objective C mode for Emacs. The author has encountered versions of Emacs configured to use the Objective C mode for .m files and other versions of Emacs configured to use the MATLAB mode for .m files.

The Emacs user can explicitly switch to the MATLAB mode by typing Meta-X where the mysterious Emacs Meta key is usually the Escape (Esc) key on most keyboards and then typing matlab-mode at the Emacs command prompt.

If the matlab mode is not preinstalled in the user’s version of Emacs, one can get an Emacs MATLAB mode add on package from the web, for example at http://www.mathworks.com/matlabcentral/fileexchange/104

Emacs can be reconfigured to use the MATLAB mode automatically for .m files instead of the Objective C mode if needed.

As discussed, in addition to source code, it is often necessary to share data between Octave and other programs. Simple human readable file formats such as the common tab delimited file format or simple uncompressed binary file formats such as Microsoft WAV audio files are often the easiest and most cost-effective way to do this. Working with files of this type sometimes requires examining the files, for example to detect or rule out errors such as non-printing characters or incorrect header information. Emacs can be used to do this.

Emacs has a couple of special modes that are useful for examining data files and sometimes source code. Emacs has a hexadecimal viewer/editor mode that can be invoked by typing hexl-mode at the Emacs command prompt. The Emacs hexl-mode is useful for examining binary file formats and occasionally human readable formats. Emacs has a special whitespace mode that displays whitespace characters such as tab, carriage return, newline, and so forth. Type whitespace-mode at the Emacs command prompt. This can useful for examining human readable data files and also source code files where the whitespace characters may be incorrect or confusing.

Making Octave Work on the Macintosh

There are a couple of gotchas to making Octave work on the Macintosh platform. By default, on MacOS X, Octave is not set up to display plots through the X server on the Macintosh. It is necessary to set an environment variable so that Octave can display plots. This is done by entering the following command at the Octave prompt.

Octave> setenv GNUTERM 'x11'

By default, Octave uses the gnuplot package, which it installs if needed, to display plots. This environment variable tells gnuplot to use ‘x11′, the X Windows server on the Macintosh, to display plots.

As discussed above, Octave has a large number of add-on packages available through the Octave Forge web site. By default, a number of these packages do not build and install correctly on MacOS. One gets a cryptic error. This is because the mkoctfile command in Octave defaults to building 64 bit versions on the Macintosh. This can be fixed by adding the lines:

CFLAGS="-m32 ${CFLAGS}"
FFLAGS="-m32 ${FFLAGS}"
CPPFLAGS="-m32 ${CPPFLAGS}"
CXXFLAGS="-m32 ${CXXFLAGS}"
LDFLAGS="-m32 ${LDFLAGS}"

in the file Octave.app/Contents/Resources/bin/mkoctfile-3.2.3 just after the “set -e” line. The problem is that in Snow Leopard (code name for a recent version of Mac OS X) compilers try always to build in 64bit, while the libraries shipped with Octave are 32 bit. The -m32 flag forces the compiler to build 32 bit programs.

Screen Capture with Octave

Using Octave or similar programs often involves making plots, histograms, and other graphics that the user wants or needs to share with others, attach to e-mails, embed in publications, and so forth. One quick way to do this on Microsoft Windows and Macintosh platforms is through the built-in screen capture capabilities of the operating system and the gnuplot package used by Octave.

On Microsoft Windows, there is a “Print Screen” button on keyboards that will capture the entire screen to the MS Windows clipboard. One can then use GIMP or many other image editors to create a file from the contents of the clipboard. One can then edit the image file to select desired regions and so forth. On MS Windows, the gnuplot display window has a window capture feature. One can type Control-C in the plot window to capture the plot window to the Windows clipboard. One can also select the plot window menu by clicking the icon in the upper left corner of the plot window, then “Options” from the menu, and then “Copy to Clipboard” from the Options menu.

The Macintosh has a number of keyboard shortcuts to capture either the entire screen or a region of the screen to the clipboard or to a file on the Macintosh desktop. These keyboard shortcuts are (Apple Command)-Shift-3 to capture the entire screen to a file on the Desktop, (Apple Command Key)-Shift-4 to capture a region to a file on the Desktop, (Apple Command Key)-Control-Shift-3 to capture the entire screen to the Macintosh clipboard, and (Apple Command Key)-Control-Shift-4 to capture a region to the Macintosh clipboard. The region commands turn the mouse cursor into a cross-hairs icon; the user can select a region to capture by clicking and dragging on the screen with the cross hairs.

Conclusion

Octave is free, both free as in beer and free as in speech. Octave has many virtues, notably that it is mostly compatible with MATLAB, which is currently the de facto standard for numerical and mathematical programming. Octave can be used to quickly develop software for a wide range of practical problems including the prediction of buying patterns, image and signal processing, pure and applied scientific research, and the invention and design of new machines, to name only a few. With the tools, methods, and commands discussed above, it is possible to make Octave work quickly and cost-effectively with a wide range of other programs on all three major computing platforms: MS Windows, Linux/Unix, and Macintosh.

About the Author

John F. McGowan, Ph.D. is a software developer, research scientist, and consultant. He works primarily in the area of complex algorithms that embody advanced mathematical and logical concepts, including speech recognition and video compression technologies. He has extensive experience developing software in C, C++, Visual Basic, Mathematica, MATLAB, and many other programming languages. He is probably best known for his AVI Overview, an Internet FAQ (Frequently Asked Questions) on the Microsoft AVI (Audio Video Interleave) file format. He has worked as a contractor at NASA Ames Research Center involved in the research and development of image and video processing algorithms and technology. He has published articles on the origin and evolution of life, the exploration of Mars (anticipating the discovery of methane on Mars), and cheap access to space. He has a Ph.D. in physics from the University of Illinois at Urbana-Champaign and a B.S. in physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11@earthlink.net.

Possibly related articles:

Octave: An Alternative to the High Cost of MATLAB

John F. McGowan, Ph.D. — Mon, 17 Jan 2011 18:00:26 +0000

Introduction

Octave is a free (both free as in beer and free as in speech) MATLAB compatible numerical programming tool available under the GNU General Public License. MATLAB, an abbreviation for “Matrix Laboratory”, is currently the most widely used commercial, proprietary numerical programming tool. Even a single license for the MATLAB software is a substantial cost. MATLAB is essentially a scripting language similar to Perl, Python, or PHP with a comprehensive, highly integrated set of numerical, statistical, mathematical, and graphical functions including, for example, the Fourier transform, matrix inversion, and histograms. In part because MATLAB has become the de facto industry standard for numerical programming, Octave is of particular interest to individuals, companies, and organizations engaged in numerical and mathematical programming and research and development.

A Little Bit About MATLAB

MATLAB was originally developed in the 1970′s as an easy-to-use interactive front-end for recently developed Fortran libraries of matrix algebra and matrix manipulation functions. For this reason, MATLAB was originally designed around two-dimensional matrices of real numbers. In the 1980′s, MATLAB was rewritten in the then popular C programming language and converted into a commercial software product from MathWorks. Over the years, MATLAB has been extended to support multi-dimensional numerical arrays, character strings, many kinds of mathematics beyond matrix and linear algebra, object-oriented programming, and many other features. MATLAB has a number of good features, but it is also a hodge-podge of features added by many different programmers at MathWorks over the last 30 years.

Octave

Octave was developed by John W. Eaton and others as a free clone of MATLAB. Octave shares most of the syntax and features of MATLAB. For a wide range of mathematical and numerical programming tasks, Octave is excellent and can do everything MATLAB can do. It is available as both source code and pre-compiled binaries for the Microsoft Windows, Mac OS X, and Linux/Unix platforms. Octave can be easily downloaded from the Octave web site: http://www.gnu.org/software/octave/

Octave and MATLAB have almost identical syntax. There is extensive information on the Web about MATLAB and Octave as well as many books about MATLAB. This article will focus on certain specific issues and features deemed especially important in adopting or using Octave for numerical and mathematically oriented projects. For detailed information on the syntax, readers are referred to the extensive and excellent on-line and published documentation.

Example 1: Adding Two Vectors in Octave

octave-3.2.4.exe:10> a = [1 2 3]
 a = 1  2  3

octave-3.2.4.exe:11> b = [4 5 6]
 b = 4  5  6

octave-3.2.4.exe:12> a + b
 ans = 5  7  9

Example 2: A Numerical Range in Octave

octave-3.2.4.exe:13> 5:10
 ans = 5   6   7   8   9  10

Example 3: A Simple Plot

octave-3.2.4.exe:15> a = 1:100;
octave-3.2.4.exe:16> pi
ans = 3.1416
octave-3.2.4.exe:17> data = sin(2*pi*a/20);
octave-3.2.4.exe:18> plot(data)

Octave Plot

Octave includes built-in functions to read and write human readable tab-delimited and comma separated values (CSV) data files. In this day and age, most computer programs ranging from databases to spreadsheets to word processors can easily read and write data files in tab-delimited or comma separated value formats. Almost all computer programming languages can easily read and write tab-delimited data files and, with somewhat more work, comma separated value data files. The Octave built-in functions for this are: dlmread, dlmwrite, csvread, and csvwrite. In real world projects, it is actually rare to use Octave or similar tools fully standalone. One is usually importing data from other programs and exporting data to other programs. With huge disk drives and powerful modern computer networks, it is nearly always possible to use human readable tab-delimited or comma separated value data files to import, export, and share data. This is often the most convenient way to integrate Octave with other tools. In addition, Octave can read and write binary data, for example through C style built-in fopen, fread, fwrite, and fclose functions.

Octave’s support for common multimedia and binary file formats in somewhat limited but adequate. Octave can read and write a number of binary image file formats using its imread and imwrite functions. The author works a great deal with digital audio, including audio special effects and speech recognition. Octave can read and write the widely used Microsoft WAV (Wave) audio file format using the wavread and wavwrite functions. Uncompressed wave audio files are quite simple and can be easily read and written at a low level by almost any programming language as well as many common audio processing tools on all of the major computer operating systems: Microsoft Windows goes without saying, both Mac OS and the iPhone iOS support wave files, and many variants of Linux and Unix. The popular free (as in beer, as in speech) Audacity audio processing program can both read and write uncompressed WAV audio files; Audacity is available as both source code and pre-compiled binaries for Windows, Mac, and Linux platforms.

Octave is quite fast. The basic data type of Octave is a “matrix,” which now means a multi-dimensional array of double precision floating point numbers. Today, there is rarely a substantial difference in speed between single and double precision floating point arithmetic on most platforms. The various matrix operations built into Octave are very close in speed to fast compiled code. Because its basic data type is basically a multi-dimensional array of native data types (double precision floating point) Octave (and MATLAB) can generally perform faster on numerical programming tasks than list-based symbolic manipulation programs such as Mathematica. Octave also supports a simple mechanism to add fast compiled C functions to Octave through its mkoctfile function/command:

mkoctfile [-options] file ...

The mkoctfile function compiles source code written in C, C++, or Fortran. Depending on the options used with mkoctfile, the compiled code can be called within Octave or can be used as a stand-alone application. mkoctfile can be called from the shell prompt or from the Octave prompt.

Because Octave is free and available for download on all major computer platforms, it is now easy to create mathematical or numerical scripts to share with anyone on these platforms (MS Windows, Mac OS X, or Linux/Unix) without the huge cost of the MATLAB program.

Caveats

While Octave has very extensive features — few users will need more — Octave does not support all of the advanced and recent features of MATLAB. While Octave is free, it is licensed under the GNU General Public License (GPL), rather than the Berkeley style licenses that allow users to incorporate source code into their proprietary commericial programs. The GPL is sometimes a problem for commercial users of Octave or other free software. Octave error messages can be cryptic.

Both Octave and MATLAB have little support for symbolic manipulation. In many areas such as signal and image processing, this is rarely a concern. The author has used the free Maxima (formerly MACSYMA) program successfully for symbolic manipulation tasks that Octave could not perform.

Alternatives

There are many free and other not-so-free numerical, mathematical, and symbolic programming tools including MATLAB, Mathematica, MAPLE, Sage, Python/NumPy, Axiom, Maxima, and many others. Interested readers are referred to many online sources and discussions for a comprehensive list of alternatives. The special advantages of Octave are that it is mature software, has a large user base, is free, is available in both source code and pre-compiled binaries for all three major computer platforms, and is mostly compatible with MATLAB, which has become the de facto industry standard for numerical programming in recent years.

Conclusion

For many uses and users, Octave is an excellent alternative to MATLAB or an excellent complement to MATLAB. Some users may find that they will want one licensed copy of MATLAB to make sure their scripts will run under standard MATLAB while using Octave on all of their computers to save costs and to ensure their scripts can be run by anyone, anywhere at no cost using the free Octave tool. Many users will be glad to entirely save the considerable cost of MATLAB, no small amount especially in the current difficult economy.

About the Author

Recommended Octave books

Sponsor’s message: Check out Math Better Explained, an elegant and insightful ebook and set of screencasts that will help you see math in a new light and experience more of those awesome “aha!” moments when ideas suddenly click.

Possibly related articles:

Improve your math and programming skills with Project Euler

Antonio Cangiano — Wed, 19 Aug 2009 05:11:06 +0000

Project Euler can be an addictive site for people who are passionate about mathematics and programming. You don’t need to be an expert in either of these two disciplines to participate in it’s challenges, but any such skills you may already possess will be an asset.

The author of the site, describes the project as follows:

Project Euler is a series of challenging mathematical/computer programming problems that will require more than just mathematical insights to solve. Although mathematics will help you arrive at elegant and efficient methods, the use of a computer and programming skills will be required to solve most problems.

When you first register, you will be presented with 252 problems available to be solved. This number changes frequently, however, as new problems are routinely created and added to the list. The solutions to some problems veer towards the obvious, while others require advanced mathematics to tackle. Likewise, a few can be solved with paper and pencil and a good dose of insight. Most however, will require the aid of a computer program written in the language of your choice.

A sample problem (187)

Each program you execute should output your result within a minute. That’s part of the challenge, even though there is no way for the site to enforce this rule.

When you think you’ve found the correct result for a given problem, you can fill in the result field for the problem, and see if you got it right. If you have, your answer will be added to the list of problems you solved and will count towards your overall score, while bumping up your position on the participant scoreboards. There are several scoreboards, depending on what level you are at, which in turn is determined by the number of problems you have solved so far.

More importantly, solving a problem will gain you access to the respective forum (for that problem), where you’ll be able to see solutions and discussions from other participants, who may have used different languages and solution strategies.

Project Euler is appealing for a variety of reasons. In particular:

It allows you to explore areas of math that you might not be familiar with. Solving certain problems may require you do a fair amount of research and in turn help you to learn more about certain branches of mathematics;
If you are a math major who is trying to hone their programming skills, you’ll find Project Euler to be challenging, fun, and an excellent opportunity for improvement. Having to implement several small programs and being able to compare your style with those of other people in the forum, will definitely end up improving your knowledge of many algorithms and your skills as a programmer;
The site’s one minute rule has an important consequence. The computational complexity of your programs really ends up mattering. While some simple problems can in fact be brute forced, the majority of them require you to write faster algorithms and take advantage of mathematical insight so as to improve the performance of your program. For certain problems a naive approach would take more than a lifetime on current PCs;
Finally, if you are a programmer who’s scoping out a new programming language, be it Python, Ruby, Scala, Haskell or Erlang, you’ll find a great ally in Project Euler. Having to write hundreds of programs in a given language, will naturally increase your familiarity with that language. And again, comparing your newcomer coding style with those of more experience participants, will no doubt contribute to your advancement within the given language you’re focusing on.

Project Euler truly deserves five stars. I recommend it highly to anyone who is unafraid of writing a few lines of code in order to solve interesting math problems.

Possibly related articles:

Complex Algorithm Research and Development: Harder Than Many Think

John F. McGowan, Ph.D. — Mon, 20 Jul 2009 17:57:31 +0000

An algorithm is a procedure or formula for solving a problem. For the purposes of this article, a complex algorithm is defined as an algorithm that embodies advanced mathematical or logical methods and requires at least one thousand (1000) lines of the C/C++ programming language to implement. The term C/C++ is used intentionally to reflect the reality that object-oriented methods are of limited use in complex algorithms, although the algorithms are often packaged inside an “object” for easy integration into applications.

Complex algorithms are typically implemented as either computer software or in custom VLSI chips (e.g. Application Specific Integrated Circuits or ASICs). Research and development of complex algorithms is a specialized area and differs in a number of ways from most software and hardware development. Remarkably, many computer software business and engineering professionals often underestimate or do not understand the difficulties and scope of complex algorithm projects.

Complex algorithms are already in widespread use in commercial applications. Prominent examples include the video compression algorithms that enable BluRay, DVD Video, YouTube, and many other modern digital video systems. The US DVD Video market is around $25 billion per year (2007). Although limited, speech recognition such as now frequently encountered in telephone help and customer service systems is another example. Other examples include encryption, seismic modeling used in oil and gas exploration, sophisticated financial models, traffic models, and many others.

Complex algorithms may solve a range of major problems confronting the human race including major diseases such as cancer, the need for more and cheaper energy, and so forth. Molecular modeling may enable the design of drugs or systems of drugs that can selectively target and destroy cancer cells based on the identifying characteristics of cancer cells such as chromosomal anomalies, something currently impossible. Electromagnetic modeling software may enable the successful design and fabrication of working commercial fusion power sources. These are potential trillion-dollar markets. The global annual energy market is over $1 Trillion.

Note: Since I am a software developer, I will focus primarily on complex algorithms developed and implemented as computer software. Many of my comments apply equally well to hardware implementation. Where I have some knowledge and experience, I will make some comments on specific hardware issues.

Complex Algorithms Are Difficult

Complex algorithms are usually quite difficult to develop and often take longer than experience with other software projects would indicate. Although there are exceptions, complex algorithm projects usually take between four (4) months and several years. True research projects in which new mathematical or logical methods are developed are extremely unpredictable and typically take years. Most major scientific discoveries and inventions have taken at least five (5) years.

Complex algorithms frequently involve a tight coupling between different parts of the algorithm. All parts must work together within tight tolerances for the entire algorithm to work. This is similar to mechanical systems such as automobile engines or mechanical clocks. Indeed, implementations of complex algorithms are often referred to as “engines”, partly for this reason. Complex algorithms are often very unforgiving. Even very small errors, getting a single bit wrong, results in the implementation failing. This occurs frequently with encryption where usually every bit must be correct and video compression where even small errors often result in unacceptable “artifacts” in the decoded video. In practical terms, this means that the amount of time spent per line of working code is often significantly larger for complex algorithm projects than other software projects such as web sites, user interfaces, database reporting systems, and so forth.

Most commercial software projects involve at most mathematics taught in early high school (9th, 10th grade) in the US. Even advanced high school mathematics such as the solution to quadratic equations is uncommon outside of computer graphics. Complex algorithms in widespread use today typically involve mathematics that is taught in the first and second year of college at a good college or university in the US. A few complex algorithms involve more advanced mathematics. For example, the Global Positioning System (GPS) uses General Relativity, advanced undergraduate or graduate level mathematics, to determine the location and time correctly. In the future, more advanced mathematics may be needed for pattern recognition and other advanced tasks. Most commercial software developers do not have much experience with mathematical software at the level found in complex algorithms. Research and development of complex algorithms often requires a range of mathematical and logical skills that are not common.

Return on Investment

The return on investment for a successful complex algorithm project can be very high. Complex algorithm research and development is typically done by small teams or individuals. Small teams are the most common. Even a multi-year project, for example five years, with a ten person team (a large team) has a total cost of about $7.5 million (using a total cost per full time employee of $150K/year). A home run can solve a billion dollar or larger problem, bringing in hundreds of millions or even billions of dollars.

Return = $100 M / $7.5 M = 13.3 (small home run)
Return = $1 B / $7.5 M = 133 (big home run)
Return = $1 T / $7.5 M = 133,000 (off the charts)

The greatest opportunities and the greatest risks lie in areas that require development of new mathematical or logical methods; that is true research. New complex algorithms can be converted very rapidly to commercial software products, even in a matter of months, as happened with new video compression algorithms in 2003.

Research and Development

The commercial software industry focuses overwhelmingly on “technically feasible” projects. Many venture capital firms explicitly claim to only invest in proven, technically feasible projects. Similar thinking pervades the commercial software industry. Where complex algorithms are concerned, technically feasible means proven algorithms for which working prototypes exist somewhere. The working prototypes are usually computer programs, often slow, that successfully implement the algorithm. These are frequently prototypes in the C or C++ programming languages, although Java is becoming more common (see the discussion of software engineering below). Thus, most commercial projects in the complex algorithms arena involve such tasks as porting algorithms to a different platform (for example, Unix to Windows), optimizing the algorithms for a new platform, integrating the algorithms into an application program such as a media player, converting a prototype into a production system, and so forth. Most research scientists would call these activities “Development” and not “Research” or “Research and Development”.

The commercial software industry follows a widely accepted rule to avoid projects that are not technically feasible, meaning true research projects. Nonetheless, the rhetoric of the commercial software industry, both aimed at unsophisticated investors and customers is the opposite. Terms like research, science, and research and development are used routinely to describe commercial software development activities. Many companies make statements that either explicitly claim or imply that the company has a large R&D group engaged in true research. Note that rhetoric aimed at sophisticated investors such as venture capitalists is often the opposite, which can be quite confusing.

Historically, the commercial software industry has relied heavily on government sponsored research programs such as the Defense Advanced Research Projects Agency (DARPA) and the National Aeronautics and Space Administration (NASA) for the true research in software. Many types of software and specific software products can be traced back to government sponsored research programs. Some well known examples include the Internet, originally a DARPA project, and the World Wide Web, which grew out of research projects at CERN and NCSA. Many other examples exist. Essentially all speech recognition software is derived from research sponsored by DARPA, especially projects at Carnegie-Mellon University. Nonetheless, industry rhetoric often invokes the image of private inventors in garages, the Wright brothers, and similar images of “free enterprise” and individual initiative. Typically, the putative inventor such as Tim Berners-Lee or Marc Andreessen is emphasized and the relevant government research program ignored or downplayed. Often there is little or no progress in commercial software if the relevant government research program is unable to make progress. This is most evident in pattern recognition and artificial intelligence, where progress has been very slow or non-existent.

Many government research programs are afflicted by a single “right way” that is pursued to the exclusion of all others. If this right way is good, then there is steady progress in the associated commercial software field. DARPA in particular relies upon periodic contests pitting different methods against one another. This has repeatedly resulted in a single approach that showed early promise taking over a field. A contest of this type during the 1970’s resulted in the so-called Hidden Markov Model (HMM) based speech recognition approach replacing essentially all speech recognition research on a global scale. Essentially all major speech recognition research groups, many directly funded by DARPA, pursue some variant of the HMM algorithm. Yet the performance of the HMM algorithm continues to be quite limited after 30+ years.

Most major commercial opportunities in complex algorithms require a company to fund and undertake genuine research, a difficult task that few companies understand. Artificial intelligence, speech recognition, cures for major diseases such as cancer or working fusion energy sources require substantial research.

Evaluating Technical Feasibility

In complex algorithms, technical feasibility generally means that a working prototype of the complex algorithm exists. In practice, working prototypes are often implemented in computer programming languages such as ANSI C, C++, or Java. Special algorithm research and development tools such as Matlab and Mathematica are also used. The working prototype may be slow, inefficient, or have other limitations, but it is or should be a proof of concept.

Seemingly, it should be easy to evaluate technical feasibility. Simply acquire and test the working prototype. Nonetheless, there is a long history of ostensibly sophisticated organizations and investors investing in complex algorithm projects that are not technically feasible, often apparently in the belief that the relevant problem had been solved. This has happened repeatedly with pattern recognition and artificial intelligence. It has also happened with various data compression algorithms.

For some reason, some organizations and investors do not even acquire and test the working prototypes. This is absolutely essential. It is often said that one should invest in people, not in ideas. Venture capitalists and other sophisticated investors often quote this platitude. However, the history of complex algorithms is filled with companies and projects with officers, directors, managers, chief scientists, and so forth with gold-plated resumes, impressive credentials, top degrees from top universities, and so forth that have flopped. The bottom line in complex algorithms is that the ideas count. If the complex algorithm doesn’t work, it doesn’t work. The company or project will fail for purely technical reasons. This is the lesson, for example, of the Pen Computing fad of the early 1990’s, GO, and Lernout and Hauspie (see below). This is also one of the lessons of the complex financial models involved in the current global financial crisis (2008). Due diligence requires a thorough, well-considered independent test and evaluation of the working prototype.

It can be difficult to evaluate technology demonstrations of complex algorithms. Often there are subtle and not so subtle ways to manipulate a technology demonstration. For example, in the early days (1995) of MPEG digital video on personal computers, MPEG software video player companies gave technology demonstrations on high-end personal computers with high-end video acceleration graphics cards. In pattern recognition, many present-day (2008) pattern recognition algorithms can achieve recognition rates in the range of 80-95% at least under certain controlled conditions. This seems high. In schools, 80-95% is usually a B or an A, a good grade. However, practical real-world pattern recognition often requires a recognition rate of 98 or 99% or even better under field conditions. In both examples, the technology demonstration can be misleading without demonstrably false statements.

Actual fraud can be difficult to detect and difficult to prove. It is important to conduct tests and evaluations of working prototypes independent of the purveyor or other interested parties who may have a reason to perpetrate a fraud. Complex algorithms can involve the potential for very large returns on investment—as noted above. Consequently, there is considerable motivation for fraud in some cases.

How Does Research Differ from Development?

Commercial software development is usually unpredictable. Software projects frequently involve unexpected problems and usually take substantially longer than planned. Nonetheless, technically feasible commercial software development projects are more predictable than true research projects. Often if one takes a conservative cost and schedule estimate and multiplies this by a factor of three to four, one gets the actual cost and schedule of the project. A common joke with a great deal of truth is: to get the real schedule multiply the official schedule by PI (3.14) for running around in a circle. Because the project is technically feasible it can certainly be completed. Massive cost and schedule overruns (such as factors of ten) can usually be explained by incompetence or severe political problems. True research is extremely unpredictable. Many true research projects simply fail. The researchers are unable to find the solution. For example, to date (2008), essentially all attempts to decipher human speech have failed in close to a century of attempts. Substantial research efforts at Bell Labs, MIT, and other institutions have failed to determine why certain sound spectra correspond to the different sounds in English and other languages. Even in successful research, estimates are often way off. For example, the mathematician Johannes Kepler made a bet in 1600 that he could determine the orbit of Mars in eight (8) days. His discovery of the elliptical orbit of Mars and other planets, one of the most important and difficult discoveries in scientific history, took five frustrating years in which every attempt to solve the problem failed until he found the answer in just a few days in 1605. This process in which long periods of little or no progress are punctuated by sudden unpredictable leaps forward is typical of true research, especially major scientific discoveries or technological inventions.

True research, especially major scientific discoveries or inventions, usually involves a very large amount of trial and error. Often, after many failures, there is a leap or leaps in which a new approach or concept is tried which unexpectedly solves the problem. Most major scientific discoveries or inventions took somewhere between five (5) and twenty (20) years. This is significantly longer than the time frame of typical commercial software industry and venture capital funded projects. In many cases, one is talking about five to twenty years of failure followed by a “breakthrough”, as in Kepler’s case.

In my experience, people involved in commercial software development are often unaware that they have little or no experience with true research. The misleading rhetoric of the computer software industry often leads people involved in commercial software development to think that they are engaged in the sort of true research conjured up by iconic names like Einstein or the Wright brothers. This undoubtedly leads to many bad decisions and frustrating experiences.

The Importance of Rapid Prototyping

History records many remarkable instances when an individual or small team succeeded in making a major scientific discovery or invention on a very small budget, sometimes beating far better funded competitors. Major scientific discoveries or inventions almost always involve a large amount of trial and error. Discoverers or inventors who managed to make a major discovery or invention on a shoe-string budget usually found a very fast, inexpensive way to perform the many trials and errors required to make a major discovery or invention.

For example, James Watt is remembered for inventing the separate condenser steam engine, a major conceptual leap that turned the steam engine from a niche device used in coal mining to a major driver of the industrial revolution. The Newcomen steam engines of Watt’s time were huge expensive house-sized engines. Watt however built and experimented with tiny scale models built from inexpensive wood, copper, and other materials. This enabled him to perform hundreds of trials and errors that led to the breakthrough concept of the separate condenser that radically improved the Newcomen steam engine.

Octave Chanute and the Wright brothers, his proteges, conducted research and development of gliders constructed of cheap wood and canvas. The gliders were flown at low altitude on soft sand beaches, first in Gary, Indiana near Chicago where Chanute lived, and later at Kitty Hawk in North Carolina. This meant that the inevitable damage from crashes was limited and easily repaired. The pilots did not die from the crashes as was common with other early would-be aviators. By delaying work on the expensive engines until last (they planned to buy a commercial off-the-shelf engine), they avoided the enormous cost involved in repairing or replacing an engine after each crash. This enabled Chanute and the Wright brothers to eventually succeed where better funded efforts such as Hiram Maxim and Samuel Langley failed.

Progress in aviation and rocketry today is quite slow, almost flat-lined since 1970, in part because the cost of a single trial, especially a new high performance engine, has become extremely high, easily in the millions if not billions of dollars per prototype engine and vehicle. In rocketry and other high performance engines, the prototype engine and vehicle are often destroyed during each trial. Internet entrepreneur Elon Musk of PayPal fame encountered this problem with his SpaceX startup as have many other Internet and software entrepreneurs attracted by the dream of space travel.

In algorithm research and development today, rapid prototyping tools such as Matlab and Mathematica (see below) speed up and reduce the cost of the many trials and errors required in true research. This is very important because the number of trials and errors is usually very large.

The Importance of Conceptual Analysis

Most major scientific discoveries and inventions usually involve a large amount of conceptual analysis expressed in words and pictures (often hundreds of thousands of words). It is common to find lengthy verbal discussions of the issues combined with rough sketches or drawings of concepts. For example, Octave Chanute wrote an entire book Progress in Flying Machines containing his lengthy verbal analysis of the problem of flight. This book outlines his successful research plan to develop working powered flight. It contains several rough drawings, as is common in major breakthroughs, and only a few brief calculations. The mathematician Johannes Kepler devoted much of his book, now known as New Astronomy, to a lengthy conceptual analysis of the problem of planetary orbits which was critical to his resolution of the problem.

At some point, these verbal analyses are refined into precise technical drawings in the case of mechanical inventions and specific mathematical expressions in the case of mathematical discoveries like Kepler’s. However, the verbal and visual analysis appears to be critical in many discoveries and inventions and usually comes first. It is likely that this sort of verbal and visual analysis will be essential to solve many problems such as artificial intelligence and pattern recognition.

Historically, this conceptual analysis was considered a part of philosophy. Much of the classical training in Greek philosophy and mathematics probably provided important training in this conceptual analysis. The discovery of new mathematical expressions of practical use strictly by the symbolic manipulation and the highly abstract thought favored by the famous mathematician David Hilbert and his school at the University of Gottingen in the early 20th century seems to be rare. This is specifically mentioned because Hilbert’s extremely abstract approach to higher mathematics has come to dominate mathematics and theoretical physics in the 20th century.

Some Famous Flops

Complex algorithm research and development is a treacherous area. There have been numerous flops and fiascoes over the years. It is easy to misjudge the technical feasibility of projects. There is a long history of exaggerated claims for complex algorithms that emulate aspects of human intelligence such as speech recognition. There has been enormous success in data compression over the last few decades. Nonetheless, there is a long history of exaggerated claims for advanced in data compression. Video and other data compression involves complex algorithms that are difficult to evaluate. Caveat emptor!

The Pen computing fad of the early 1990’s is an example of a famous flop. The most prominent of these firms was Jerry Kaplan’s GO, described in his book Startup. GO and similar firms’ business plans hinged on handwriting recognition, an unsolved problem in pattern recognition. Kaplan actually devotes only a few pages of his book to the handwriting recognition problem.

Another notorious example is the speech recognition firm Lernout and Hauspie. Lernout and Hauspie collapsed in a major financial scandal with court cases and allegations of massive fraud. Again, the success of Lernout and Hauspie’s business depended on the solution of the speech recognition problem, which remains largely unsolved even today.

Note that many apparently sophisticated investors invested many millions of dollars in both GO and Lernout and Hauspie, even though a modicum of research would have revealed the poor state of handwriting and speech recognition technology at the time.

Some Famous Successes

Video and audio compression is one of the most successful areas in complex algorithms. Technologies such as VideoCD, DVD, MP3, and BluRay all incorporate sophisticated audio and video compression algorithms.

A major breakthrough in video compression reached the market in 2003, embodied in H.264, Windows Media 10, Flash Video, and other video standards and products. Prior to 2003 the bitrate for usable, loosely VHS quality video was about one (1) megabit per second. In 2003, the new video technologies achieved a bit rate of around 275 Kilobits/second, often with close to DVD quality with proper tuning of the compression. This was a truly major advance, a rare technological leap forward. This enabled YouTube and other forms of Internet/web video over DSL connections.

The bottom line is that complex algorithm research and development can be done, but it is difficult.

Software Engineering

There are significant differences between software engineering for complex algorithm research and development and mainstream software development. As mentioned above, complex algorithms often involve a tight coupling between parts of the algorithm that makes development more difficult and tedious than most software development.

It is often easier to research and develop complex algorithms using tools such as Matlab, Mathematica, AXIOM, or Maxima (formerly known as MACSYMA). These are scripting languages similar to Python or PHP. They usually have implicit variable declaration and/or conversion. They are usually “weakly typed” languages and break many textbook rules of “good” software engineering. They include comprehensive, well-integrated libraries of mathematical, numerical, and statistical functions. They usually have a data type known variously as a list, vector, or matrix that represents sequences of numerical or symbolic data in a flexible way. These tools are sometimes referred to as computer algebra systems (CAS), although this is really only one subset of their features.

Adding Two Vectors in Mathematica

A = {1.0, 2.0, 3.0}; (* A is a Mathematica list *)
B = {1.1, 0.0, 4.0};
C = A + B
Out[1]={2.1, 2.0, 7.0}

Adding Two Vectors in C/C++

#include 
double A[3] = {1.0, 2.0, 3.0}; // A is a C++ array
double B[3] = {1.1, 0.0, 4.0};
double C[3];
int index;

for(index = 0; index <3; index++)
   C[index] = A[index] + B[index];

cout << “{“ << C[1] << “,” <<
C[2] << “,” << C[3] << “}” <<
endl;

Note that there is a vector class template in the C++ Standard Template Library (STL) with somewhat similar properties to the lists in Mathematica. The above comparison is for illustrative purposes. Even using the STL classes, it is usually much easier to research, develop, and test algorithms in these tools than using traditional compiled, strongly typed languages such as C/C++, Java, or . However, these tools are slow and require large amounts of memory. This is a significant drawback. Once an algorithm is developed, it is often necessary to convert the algorithm to a fast compiled language for performance reasons. This is easier if the target fast language has good libraries of mathematical, numerical, and statistical functions.

One can also research and develop algorithms directly in a fast programming language such as C/C++ or Java. This avoids conversion costs, speed, and memory issues. However, it is often much easier to do research and development using a tool such as Matlab or Mathematica.

The leading algorithm research and development tools are:

MATLAB: Matlab is widely used in the commercial world, especially in digital signal processing.
MATHEMATICA: Mathematica is widely used in government sponsored research and development and academic research. It has a following in Wall Street finance and economics.
AXIOM: Axiom is free, open source, with a Berkeley style license. AXIOM was started in 1971 and has over 300 man years of work integrated into it.
MAXIMA: Maxima is free, open-source, with a GNU license.

Many fast programming languages have been used for complex algorithms. The most popular are probably:

ANSI C: ANSI C is almost universally available for all processors. It is simple, efficient, with small memory needs and high speed.
C++: C++ is object-oriented. It often has larger memory needs than C and can be slower.
Java: Java is compiled to byte-codes, but is approaching C/C++ in speed. It can be slower and less efficient. It can be easier to reverse engineer.

The dream algorithm R&D tool would be similar to Matlab or Mathematica but could be compiled to fast, efficient binaries similar to ANSI C and would be available for all platforms. An integrated GUI builder similar to Visual Basic and integrated network support would be helpful. The biggest single weakness of all kinds of scripting languages is that they are slow and cannot be compiled. For compute- intensive complex algorithms this can be a very significant problem. Of scripting languages, only Visual Basic 6 appears to have solved the problem of producing a compiler that can produce binary executables with similar performance to C/C++.

These algorithm research and development tools are not, of course, a substitute for thought, creativity, and the extensive conceptual analysis frequently required for major advances. Trial and error alone, without insight, rarely succeeds.

Conclusion

Complex algorithm research and development can be done successfully. Some great successes exist. Nonetheless, it is not easy and many things can go wrong. The project scope is significant. Project feasibility is difficult to assess. Genuine breakthroughs are unpredictable and take time. The return on investment for a home run can be five to thousands of times the original investment. There are great unrealized opportunities, most of which require genuine research.

About the Author

John F. McGowan, Ph.D. is a software developer, research scientist, and consultant. He works primarily in the area of complex algorithms that embody advanced mathematical and logical concepts, including speech recognition and video compression technologies. He has many years of experience developing software in Visual Basic, C++, and many other programming languages and environments. He has a Ph.D. in Physics from the University of Illinois at Urbana-Champaign and a B.S.in Physics from the California Institute of Technology (Caltech). He can be reached at jmcgowan11 (at) earthlink.net.

References

Some Complex Algorithms

MPEG compression, one of the great success stories: http://www.chiariglione.org/mpeg/
x264 is a free, opensource h.264 video encoder: http://www.videolan.org/developers/x264.html
The Carnegie Mellon Sphinx Project, an open-source speech recognition engine: http://cmusphinx.sourceforge.net
National Library of Medicine Insight Image Registration and Segmentation Toolkit: http://www.itk.org

Books and Articles

Startup: A Silicon Valley Adventure, by Jerry Kaplan, Houghton Mifflin Co, Boston, 1995, ISBN 0-140-25731-4
“How High-Tech Dream Shattered in Scandal at Lernout & Hauspie”, by Mark Maremont, Jesse Eisinger, and John Carreyrou, Wall Street Journal, December 7, 2000
New Astronomy (Nova Astronomia), by Johannes Kepler, Translated from the Latin original by William H. Donahue, Cambridge University Press, Cambridge, UK, 1992, ISBN 0-521-30131-9

This article was written by John F. McGowan, PhD. If you’d like to write for Math-Blog.com, please email us at submissions (at) math-blog.com.

Possibly related articles:

Polynomial Root-finding with the Jenkins-Traub Algorithm