Advanced Excel for Scientific Data Analysis

cgjherr writes "If the recent financial meltdown has left you wondering, 'When does exponential decay function stop?' then I have the book for you. Advanced Excel for Scientific Data Analysis is the kind of book that only comes along every twenty years. A tome so densely packed with scientific and mathematical formulas that it almost dares you to try and understand it all. A "For Dummies" book starts with a gentle introduction to the technology. This is more like a "for Mentats" book. It assumes that you know Excel very well. The first chapter alone will have you in awe as you see the author turn the lowly Excel into something that rivals Mathematica using VBA, brains, and a heaping helping of fortitude." Read on for the rest of Jack's review.

Advanced Excel for Scientific Data Analysis
author	Robert de Levie
pages	700
publisher	Oxford Press
rating	9
reviewer	Jack Herrington
ISBN	9780195370225
summary	Use Excel for high end scientific data analysis akin to Mathemetica

When I first opened this book my mouth just dropped. It had been years since I had seen a book typeset using LaTeX. But in an instant it made sense as the book is crammed packed with the kind of equations that would have been a nightmare to build with any other tools. Chapter after chapter has everything a really smart person needs to do curve fitting, statistical measures, differential equations, time-frequency analysis. But don't expect a play by play here. You will get the equations, set within a few dense paragraphs, with maybe a spreadsheet and a chart or two to show the results.

The first chapter concentrates on the getting the most out of Excel as a tool. All the chapters that follow dig into specific data analysis techniques. Chapters two, three and four are on least squares. Chapter five and six cover the analysis in the time domain including fourier transforms. Chapter seven covers differential equations. Chapter eight returns to Excel by digging in deeper into macros. Which leads into chapter nine, where we dig deeper into basic mathematical operations. Chapter ten covers matrix operations. And chapter eleven wraps it all up by giving you some spreadsheet best practices.

In University style there are also some exercises that you can do along the way if you want to tweak your brain pan a little more. To amuse myself I tried a few and I believe the book would have assessed my attempts 'wanting' if it had a voice to tell me.

Where most books like this would have several authors this book has just one; Roberte de Levie. This means that the tone, style and quality of the book is consistent throughout. A fact that you will come to appreciate as the book wades in ever increasingly deep data analysis concepts as the chapters roll on.

Though I would have preferred the book to have code samples in C#, I understand that the language of Excel is VBA and I guess I have to live with that. Thankfully VBA has come a long way and if you so inclined it would likely be easy to translate the code into C#, Java, or whatever else you like.

The fact that one person wrote the book left me wondering, "Who is this guy?" In my minds eye I kinda of figured he would look like one of those pulsing brain guys from Star Trek. Turns out he is a professor at Bowdoin College. And his fields of study include ionic equilibria, electrochemical kinetics, electrochemical oscillators, stochastic processes, and a whole lot more stuff that almost seems made up to sound impressive.

When this book isn't serving as an amazing reference for both Excel, scientific problem solving, or just insane equations it serves other purposes as well. It's a handy portable IQ test, as the count of pages you can grind through in one sitting, plus 90, is roughly your intelligence quotient. And if you fail at that you can always put a copy of the book, along with the Orange Bible, under your pillow and try to osmose your way to becoming the Kwisatz Haderach.

In all seriousness, this is a great book. It represents the kind of in-depth work and research we used to see in books that came out twenty years ago. Robert is to be applauded for his work. This is an excellent resource for anyone looking to do scientific data analysis but who was unaware of the powerful capabilities that Excel provides that is likely waiting just one Startup menu click away.

The book is not without fault. I would have preferred that it had been in color, or at least have one color section to show some of the more impressive visualizations that I'm sure would look great in color. In addition the index is silly short for a book that clocks in at 700 pages. But those are only minor quibbles for what is all-in-all an amazing piece of work.

You can purchase Advanced Excel for Scientific Data Analysis from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

alternately....

Don't do it! [burns-stat.com]

...there's a better solution

Python for scientific analysis [scipy.org],

Python is the solution I recommend for everyone who looks for tips on advanced Excel uses. Excel is OK if you just want some quick and dirty solution for a small problem, but if you have to go to the trouble of reading a book, Excel is clearly not the best solution.

For scientists and engineers who need something more than what Excel (and possibly Matlab) offers, I recommend starting with either A Byte of Python [swaroopch.com] or Dive Into Python [diveintopython.org].

Re:alternately.... kind of begs the question...

What in the world are you talking about? :)

LaTeX is a markup language. You can express math with it, but it doesn't do anything for you in terms of analysis.

Excel is good for small data sets and quick looks at stuff - but painful to develop in.

Mathematica requires college-level calculus and linear algebra... not PhD stuff by any stretch.

Anyway, you left out Matlab - which is pretty awesome. Depending on what you are doing, there is also R, Maple, Minitab, MathCAD, yada, yada, yada. Lately I've been doing stuff in Python... SAGE is pretty nifty, and the NumPy/SciPy stuff is coming along well (it is included in SAGE).

Re:alternately.... kind of begs the question...

don't forget Octave [gnu.org]! (it's more-or-less a FOSS Matlab clone, and follows more closely to Matlab syntax than SciLab)

incongruous

There's something hard to reconcile about the reviewer's obvious awe and the fact that the book was written by someone who thinks doing meaningful scientific data analysis in Excel is a good idea.

Re:incongruous

Why isn't it a good idea, and does this apply equally to OpenOffice?

Re:incongruous

Why isn't it a good idea, and does this apply equally to OpenOffice?

It's OK for simple stuff, but trying to do something like implementing a loop in a spreadsheet. And yes, the criticism applies to OO as well.

There are very good packages out there - some open source - for doing scientific analysis. I'd recommend R or Octave (a matlab clone), personally. Also, Python + NumPy + SciPy + Pylab is great for doing Matlab-like things, and it's all free as well.

Re:incongruous ...I KNEW there was a reason

openorifice... if only we could come up with a new graphic for their logo...

Re:incongruous

Well, I do put faith into my fortran compiler.

Wrong Tool

Talk about the wrong tool for the job. If you need to do any sort of serious data analysis, use R, not Excel.

eh?

"The first chapter alone will have you in awe as you see the author turn the lowly Excel into something that rivals Mathematica using VBA, brains, and a heaping helping of fortitude."

Then why not just use Mathematica?

Re:eh?

"The first chapter alone will have you in awe as you see the author turn the lowly Excel into something that rivals Mathematica using VBA, brains, and a heaping helping of fortitude."

Then why not just use Mathematica?

1. you want to interact directly with excel data you receive
2. you need to give the results to someone w/out mathematica
3. a license of mathematica costs $2500, vs $150 for Office Home and Student

Re:eh?

If you're going to mention that the Office costs $150 for a student version, you might as well mention that Mathematica's student version (identical to the full version, except for a banner upon printing) is $140.

Wrong tool for the job

Someone should tell this guy about SAGE http://www.sagemath.org/ [sagemath.org]

Spreadsheets are the right tool ...

You see, there is a fundamental problem in science and the problem can be summarized as this: how do you get the right results in order to optimize the grants that you receive. Spreadsheets are ideal for this purpose for two reasons. First of all, they are designed to handle financial data. This is great because financial data are what grants are all about. For example: will result X allow for a conference in Hawaii or California this year.

The other big reason to use spreadsheets is that they make data more maluable. Normal scientific tools make it difficult to micromanage the data that you acquire, partially because the people who produce that software have this mistaken notion that data has to be managed in a consistent way. So you're usually stuck doing the same thing to an entire dataset, and it's even difficult to treat different datasets in different way. But spreadsheets expose all of that data, so it is easy to tweak an observation here and a variable there to get the desired result to maximize your grant.

So you see, spreadsheets are a tremendously valueable tool for scientists. It is the best tool for the job.

That's nothing

turn the lowly Excel into something that rivals Mathematica using VBA, brains, and a heaping helping of fortitude

So? What's so special about that? You can turn C, Fortran, or even assembly language into something that rivals Mathematica using brains and a heaping helping of fortitude. This is arguably a better deal, since you don't need the VBA.

--MarkusQ

When all you have is a hammer...

...everything looks like a snowglobe!

Hardcore data analysis in Excel is almost always a bad idea. You can almost always find a way to do it in excel, and you can almost always find a way to do it better, faster, and cheaper somewhere else.

R, MatLab, Mathemateica, Python/Numpy, SigmaPlot, and any number of old, well written, debugged and vetted numerical libraries written in C or Fortran. I've used all of these at various times to solve something that a co-worker couldn't figure out how to do in Excel.

I fit quick linear regressions in Excel. For *anything* else, there is a better choice.

Excel is a horrible tool

for scientific data analysis.

I know it is popular and many science and engineering faculty lazily encourage their graduate students to use it. However, something like matlab beats the crap out of excel any day. Spreadsheets tend to obfuscate relationships between data, require a lot more clicking (read human intervention) and waste time that could be spent thinking about the data, and are singularly unsuited for analysis of similar sets of data (a situation any scientist faces when he has to do a series of experiments).
Matlab might take sometime to initially write the scripts, but it is so powerful and extensible that no one in their right mind would want to use excel. If you are a slave to spreadsheets, get yourself a copy of Microcal Origin or Labplot.

Excel is especially unsuited to the task of preparing figures for scientific publications. The default formatting is at once wrong for the task and hard to change. Once you set your preferences in matlab (easy to do), you are set for life.

In my experience, excel is also rarely used for anything serious outside of US. Maybe its an indictment of how lazy, slow witted and easily misled our pool of talent is becoming.

Excel does not Excel

When I worked in the semiconductor industry in the late 90's, Excel nearly cost us several hundred grand. It had "helpfully" autocorrected a code in the documentation for a mask used in one of our clock buffer chip products. Had the engineers not caught this mistake in the printout, the fab of the chip would have been botched. The engineers were mad as I recall because they would change the code and Excel would change it back. If you can't prove what your tool is doing, you don't get to use it is what they taught me in engineering school.

Excel *could* replace SPSS (not Mathematica)

SPSS has now become the standard data analysis package for quantitative studies in social sciences. It's very crappy software, and it wouldn't take a whole lot of augmentation to get Excel do what SPSS does.

The problem is that social scientists don't want to mess with the internals too much, and SPSS made for them a point and click interface - in effect, they out-Microsofted Microsoft. They charge an insulting $1500/copy and completely dominate the universities, so they're making good money.

They seriously need some competition.

comparison of Mathematica and Excel VBA ..

You cannot be serious ..

"Excel 2007, like its predecessors, fails a standard set of intermediate-level accuracy tests [mathforum.org] in three areas: statistical distributions, random number generation, and estimation"

I can't believe you people...

Look at all those posts saying "Excel is not the right tool for this" or "When all you have is a hammer...". The point was not grokked by those folks.

I'll lay it out for you, plain and simple:

This book is like installing a linux kernel onto a wristwatch.

We should be marvelling at the feat, not lambasting a tool that was "hacked" to do so much more than it is normally used for. If you can't appreciate that kind of work, maybe you should just stick to appreciating fine arts.

How do you know if a book was typeset using LaTeX?

> It had been years since I had seen a book typeset using LaTeX.

The publishing industry (including my company) typesets books using LaTeX all the time. The reason you don't notice it (apart from the superior quality) is that it does its job of typesetting very well.

If this book has been typeset using LaTeX then I'm a Dutchman, or something has gone very wrong (and I'd like the author to contact me to let me know what).

Perhaps he was given faulty fonts, perhaps he was using a badly-written publisher's style, or perhaps he -- or his editor -- spent a long time making it look as bad as possible. Maybe OUP had it completely re-typeset in some other system without telling him. There are at least a dozen typographic faults in one paragraph alone, from unnecessary hyphenation to excessive word-spacing to bad math spacing, and LaTeX simply doesn't make those types of mistake unless you work very hard to introduce them manually.

As a test I screenshot a random paragraph [silmaril.ie] that I viewed in Amazon's "Look Inside" feature, and then retyped it in LaTeX [silmaril.ie] and typeset it (PDF [silmaril.ie]).

As I don't have the book (and wouldn't understand it anyway :-) I'd be interested to know where the information came from that it was typeset with LaTeX; and if it really was done in LaTeX, I'd love to know WTF kind of style files, fonts, and preamble were used.

Re:I have not read the book

...in the same way that MS paint is as capable as photoshop...

Yes, I use both. LaTeX if I have a choice, Word if I need to exchange docs with less enlightened colleagues.

Re:Anecdote about Excel

When I was a freshman in engineering school, my intro to engineering class required us to purchase a book similar to this. We were given two class periods to work with Excel, supervised by a TA. (it was considered a lab) I remember the assignment involved proving that sin^2+cos^2=1.

Proving that with Excel? How does that work? That's a trigonometry problem, and it follows from the definitions of the sine and cosine functions, and from Pythagoras's theorem. You do it with a pen and paper and you write 'QED' at the bottom. To prove it with Excel, you'd have to calculate the result individually for every possible angle, and unless Microsoft have released an update I haven't had yet then Excel doesn't have a transfinite number of available rows.

Oh, wait...

engineering school

That's dangerously close to reality. That's where they think that if something works the first fifty million times, then it's going to work every time.

Still, it could be worse. You could be in If you couldn't figure out Excel within those two class periods, it was recommended that you switched your major to business administration.

Yeah.