Book Review: Linux Shell Scripting Cookbook 173
vellorean writes "I have been reading Linux
Shell Scripting Cookbook by Sarath Lakshman, published by Packt,
for a while. While most people I know learn shell scripts themselves,
I was looking to refresh my concepts a little as well as have a
reference lying around on the table for fast access." Read below for the rest of vellorean's review.
First of all, let me remark by saying that shell scripting is
something learned more on a need basis than as a tool to solve the
main problem. People would seldom write shell scripts as standalone
programs (exceptions exist). However, what makes shell scripting
invaluable to know is the fact that knowing some tricks can save
several minutes, or hours, of work by automating and simplifying
certain tasks, generally (but not restricted to) file management and
data processing. Linux
Shell Scripting Cookbook does go quite far in pursuing this goal,
and is appropriate for both beginners who are looking to gain
dexterity in shell scripting, as well as intermediate users who wish
to polish their skills. The book also can double up as a quick
reference, though I would argue that the "Advanced Bash Scripting
Guide" would suit that more.
| Linux Shell Scripting Cookbook | |
| author | Sarath Lakshman |
| pages | 360 |
| publisher | Packt Publishing |
| rating | 9 |
| reviewer | Kumar Appaiah |
| ISBN | 1849513767 |
| summary | A book for beginners and intermediates, which introduces shell scripting and proceeds to provide several practical real-world recipes of useful shell scripts |
At the outset, the author clarifies that the focus will be on Bash. This, people may or may not like, but the fact that bash has become ubiquitous in terms of the available shells on Unix-like systems today, starting out with bash is not a bad thing to do. Besides, learning other shell scripting languages while knowing bash isn't too hard, since the paradigm remains the same.
The book is organized into chapters based more on utility than scripting concepts themselves, although the language aspects are brought onto the reader gradually. For instance, the examples in the first chapter focus more on the basic data elements (variables, arrays, functions etc.) as well as operators (for numbers, files etc.), and all the examples demonstrate simple usage of these concepts, and he further chapters build upon these in a gradual manner.
At the same time, if he reader has some familiarity with shell scripting and needs to only refresh or learn a certain concept, he/she needs to just read the relevant chapter. It is not too difficult to grasp the examples of the later chapters, provided some basic shell knowledge is assumed.
A positive trait in the presentation of this book is that it is all based on practical everyday examples which, with minor adaptation, can be used by many for their own daily tasks. For instance, there are several examples which describe searching for and processing files, which, I'd imagine, many users would want to do on a regular basis. Thus, providing realistic examples allows the book to double its utility. The language and approach used is simple and conversational, and the presentation is very clear, with each idea being described as a problem statement followed by a "How to do it" section with the actual code, and ending with a discussion of the nitty-gritties of the code. It is easy to go for a quick scan for those in a hurry, while those who with to read in more detail will not be disappointed either.
The book also covers a wide array of applications. For instance, there are examples on automating fetching web pages and processing them, demonstrations of parsing and simplifying and even some queries around databases wrapped around in shell. It also spans to utilities and tasks connected to statistics, backups, compression, version control and many more.
The book goes into a fair amount of detail in terms of describing the shell scripting concept under consideration. The examples used go into a fair amount of detail in order to describe to the user all the aspects involved in the method or command being used. The concepts described are fairly complete, and would be sufficient for the reader to use immediately or with just a little bit of fine tuning. In terms of breadth, the book covers most of the features of shell scripting while also describing the various facilities the shell provides access to in a Unix-like environment. Thus, the book does not disappoint in this front either.
In summary, probably the only thing I'd have liked to see more of is some emphasis on how to write more efficient shell scripts. Granted, most of the shell scripts described in the book are very simple and succinct, but a some words on how loops can be made better, or how to spot situations where pipes are not needed to solve a problem etc. might have been a nice addition. Some explanation of differences with dash, tcsh, zsh etc. might also have been nice, since a lot of users have different default shells. But all this isn't going to prevent me from giving this book a high rating, since it delivers quite well on the promises it makes at the beginning.
This is definitely a good book to have near your desk, and kudos to the author for having taken the effort to put it together. I would highly recommend it to the beginner and occasional shell user for a thorough read, and to an intermediate to have on his/her desk for borrowing the cool scripting ideas and applications the author has written in this book.
You can purchase Linux Shell Scripting Cookbook from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Why are there still shell scripts anyways? (Score:2, Informative)
Re:Why are there still shell scripts anyways? (Score:5, Insightful)
wow. how many ways can i answer that?
- there is no one true way
- people learn a skill and want to continue using it
- concise syntax / small footprint / fast load times
- high confidence shell scripts will run on unknown systems
- can be layered on years of useful scripts
- distrust of your options for doing work
Re:Why are there still shell scripts anyways? (Score:3)
- too lazy to learn another language or...
- not enough time to learn another language
Re:Why are there still shell scripts anyways? (Score:2)
- high confidence shell scripts will run on unknown systems
I guess it depends what you consider a "high confidence shell script". Avoiding GNU extensions to programs you run and sticking as strictly as possible to POSIX options and behavior would probably get you most of the way there, but that's a pretty big limitation.
The way I see it, the strengths of a shell for scripting are that it's an environment truly made for running other programs and interfacing them together, and it's an automation (scripting) environment that matches your interactive session (and thus it's easy to experiment, and translate your experience in day-to-day usage of the system to scripting and vice versa). But on the flip side, the fact that the shell and programs running in it don't share any common conventions for exchanging data is a crippling limitation...
Re:Why are there still shell scripts anyways? (Score:2)
Sure you can use a wrench to hammer in a nail but a hammer is much more appropriate. Modern scripting languages have concise syntax, small footprints and fast load times. There is a high confidence that such languages will be on unknown systems and they have been around for years sporting vast libraries cough... CPAN.. cough that rival and exceed most anybody's bag of shell tools.
The real point is that modern scripting languages are languages and as such offer sophisticated and proper design for engineering eloquent and useful solutions to real world problems. There's a reason why languages are designed and there are reasons for each and every bit of functionality and concepts that they put in them - even if you fail to grasp them. As such there is good reason to move to them instead of arcane shell incantations that do not promote sound software engineering principals.
That said I will whip out a quick bash script to do some research or investigative type stuff but if I ever find it being run frequently, growing bigger or being put into production I usually take the time to re-write it properly in Perl.
It amazes me that you can write all that about proper design in the same message as you're advocating Perl... :)
Re:Why are there still shell scripts anyways? (Score:5, Insightful)
Re:Why are there still shell scripts anyways? (Score:2)
Rah! A really good comment, saves time, works with all shells.
Comment removed (Score:2)
Re:Why are there still shell scripts anyways? (Score:2)
Not all systems have perl or python installed, but all of them have a shell. Lowest common denominator is a powerful thing sometimes.
Not all systems have any particular shell installed (UNIX/Linux systems have bourne shell, but nobody uses that). Lots of Solaris systems do not have bash (I assume the book is about bash).
I certainly cannot count on any UNIX/Linux shell to be running on Windows, but perl runs fine on Windows.
Re:Why are there still shell scripts anyways? (Score:5, Insightful)
Sysadmins who can't work in plain shell scripts are dangerous. What will you do if you lose your /usr filesystem ? Be stuck with no automation tools at all as you fix by hand ? Shell scripts have little or not dependencies, and can function where other languages aren't available.
More-over they are ubiquitious. Perl, python and ruby are common in Linux, but what will you do if you're on a different Unix altogether ? A sysadmin who can use shells efficiently can easily transfer his skills to solaris, hp-ux or several others (I did just that, and now I'm paid quite a lot more than a Linux sysadmin earns as a Unix sysadmin because the skillset is rarer). Of course there are other skills you'll need to learn (package management for example is vastly different between various unixes and most are frankly primitive if you are used to Linux) but the core Unix fundamentals remain the same, and if you don't know powerful and flexible shell scripting, in my book, you're no sysadmin at all.
Re:Why are there still shell scripts anyways? (Score:2)
You phrase the question differently than I would. I would ask why is perl not the default shell language.
I have a hard time understand the need for bash when perl is around. I can't quite make the same argument for Python and Ruby. THose are arguably better programming languages but not better administrative script languages. Perl on the otherhand is still very close to the common shell languages while being much more powerful. It's also faster than shell for almost every purpose where time matters. But this is achieved without making simple things difficult.
Indeed that was Larry's mantra: simple things should be simple, and hard things possible. So perl is the glue language of choice to supplant shell.
Perl's name was derived from the acronym Practical Extraction and Report Language. That's what Linux administrative scripting is all about. If you want to write a GUI word processor, use python.
Re:Why are there still shell scripts anyways? (Score:3)
You phrase the question differently than I would. I would ask why is perl not the default shell language.
Because Perl is huge and sh is tiny. Would you be able to port Perl to BusyBox without dramatically increasing the size of the executable?
Re:Why are there still shell scripts anyways? (Score:2)
Even my phone has 256MB of ram, and its replacement will have at least 1GB. Remind me again why this is a problem?
What is up with embedded devices having less than 64MBs of ram?
Re:Why are there still shell scripts anyways? (Score:2)
Because I don't want to pay for more RAM in my device just because you're too lazy to use the right tool for the job? :)
Re:Why are there still shell scripts anyways? (Score:4, Insightful)
OK, how much RAM does your router have? Your cable modem? A small NAS device? Your concept of embedded devices may be only a small fraction of what's out there.
Comment removed (Score:3)
Re:Why are there still shell scripts anyways? (Score:2)
I mean consumer embedded devices. I would probably just stick with actual compiled code on devices like that.
Re:Why are there still shell scripts anyways? (Score:2)
In my shop it is either bash or python take your pick, perl is not to be used on company systems.
Re:Why are there still shell scripts anyways? (Score:3)
I take option 3, not working there.
Python could be nice, if it ignored whitespace/indentation. Also if it had its own version of CPAN.
Re:Why are there still shell scripts anyways? (Score:2)
I keep hearing this as the #1 complaint about Python, but isn't Python open source? Why doesn't someone just modify it to ignore whitespace and use braces instead of indents? I don't get it.
The core developers do not want that. They had an opportunity - a mile wide - with python 3.x; but instead decided to implement a lot of nearly useless features.
There is also the issue of backward compatibility.
Re:Why are there still shell scripts anyways? (Score:2)
I keep hearing this as the #1 complaint about Python, but isn't Python open source? Why doesn't someone just modify it to ignore whitespace and use braces instead of indents? I don't get it.
If you create your own in-house branch of Python that behaves differently from the mainline version, then you've just given yourself more work to do - maintaining this piece of software. Presumably you'd also want to keep it up-to-date with all the changes in mainline Python, so you don't find yourself in an awkward situation years later where you're still using that in-house branch and maybe you've lost the source code with your modifications but you have tons of stuff that relies on it and not a whole lot of free time to remake your altered version or migrate your scripts or whatever....
And personally I see nothing wrong with Python's approach to whitespace.
Re:Why are there still shell scripts anyways? (Score:2)
I'm curious as to the reasons behind this. Is there some inherent problem with perl that it's banned or is it just a requirement to maintain consistency?
[John]
Re:Why are there still shell scripts anyways? (Score:2)
You phrase the question differently than I would. I would ask why is perl not the default shell language.
I have a hard time understand the need for bash when perl is around.
[snip] Larry's mantra: simple things should be simple, and hard things possible.
Perl isn't simple enough for basic scripting. Most shell scripts are little more that lists of commands with minimal control flow. For that, Perl is too verbose. The extra syntax means more places for bugs and it gets extra messy if you are generating scripts programatically. Now, I agree that more complex scripting should be done is something like Perl. "Advanced Bash" makes about as much sense to me as "Advanced duck tape application"
It's too bad that Unix shell scripting languages are so dreadful. They didn't have to be. REXX was/is an awesome scripting language for VM/CMS. Negligible syntax overhead for simple operations and you could still do complex things with a clean syntax. Perl is better for bigger things but REXX effortlessly spanned the zone from simple list to substantial program. REXX is available for Unix but it doesn't really have the same effect because the close coupling between language syntax and the default shell isn't there.
Re:Why are there still shell scripts anyways? (Score:2)
I have a hard time understand the need for bash when perl is around.
Conversely, I have a hard time understanding why anyone would use perl when bash, Ruby and Python are around.
A shell should be (needs to be) light-weight. Perl is not. Perl's syntax is horrendously complicated. Bash (and sh) syntax is not.
Re:Why are there still shell scripts anyways? (Score:2)
You phrase the question differently than I would. I would ask why is perl not the default shell language.
For one reason, you wouldn't want to have to type in something like
instead of
for everything you do.
However, the exact same syntax features that make sh good for interactive prompts make it a horrible programming language. IMO, it's tied with COBOL for the worst language still in general use. I'm familiar with a couple of dozen languages, and sh is the one I find hardest to use without constantly referring back to the documentation. I can never remember its crazy variable expansion rules and escapes, its weird logic and test operators, or its cryptic built-in variable names.
Lately, if a script looks like it's going to be longer than a single line, I use Python and import the "subprocess" module. This greatly helps my sanity.
Re:Why are there still shell scripts anyways? (Score:2)
Use IPython [scipy.org], and not only do you get an interactive Python console with tab completion, code highlighting, simple introspection, logging, macros, profiling and more, but it also acts as a shell where you can do things like ls -l, and treat the output as a Python type that you can pipe to variables and functions if you so desire. Awesome software.
Re:Why are there still shell scripts anyways? (Score:3)
Perhaps because perl's about an order of magnitude larger:
# quickpkg bash perl /usr/portage/packages/app-shells/bash-4.1_p9.tbz2 /usr/portage/packages/dev-lang/perl-5.12.2-r6.tbz2 /usr/portage/packages/app-shells/bash-4.1_p9.tbz2 /usr/portage/packages/dev-lang/perl-5.12.2-r6.tbz2
# du
1120
12372
Re:Why are there still shell scripts anyways? (Score:2)
Try running bash without grep and awk some time. bash is smaller because it calls other programs. that makes it fragile if those other programs change.
Re:Why are there still shell scripts anyways? (Score:2)
Try doing anything useful in Perl without loading modules.
And I'm saying this as a Perl fan. I use it for everything that can't be done in a 10-line shell script, but for some simple automation tasks it is just overkill.
Mart
Re:Why are there still shell scripts anyways? (Score:2)
So nothing serious, just the stuff no business can live without holding everything together. Ok got it.
Re:Why are there still shell scripts anyways? (Score:2)
If you are talking about actual commercial enterprise software that is sold for money, perl probably never had anything "serious" developed using it. It's not really a language for that.
Perl is the scripting language you use when you want more than you get with shell, but don't really care if it is optimized to the point it can be used for an enterprise app. Just like bash, perl is on every UNIX-like host out there and if you write your scripts using only perl and perl modules, the code is entirely portable to anything you can get perl to run on, including Windows. With things like PAR, you don't even need to load the modules on each host, it will create a self-extracting archive with the necessary modules and shared libraries once you archive up your script.
The other nice thing about perl is that it works quite a bit like shell already. You can run command line apps directly from perl just like you can from shell. All you have to do is use system() or the dreaded backticks. I avoid that, because it can affect portability, but if you need to upgrade an existing shell script quickly, it's easy to do. You can upgrade a shell script, whose data source is a text file, to a script that does the same thing to data stored in a database or an XML file, or on the web with trivial effort in perl and not even have to re-write much of the portion that you used in the original bash script.
Perl is always there, has modules for everything, and has been used for so long that you can find documentation for best practices to do anything with it. Ruby and Python have their uses, but they aren't solving any problems for me that haven't already been solved by perl long ago. There's simply no way to go wrong with perl if you understand it's limitations, have already taken the time to learn it and have become proficient in it. It is also possible to write readable, well-structured code with perl, they just don't force that on you.
I use:
bash = startup scripts, quick scripts where portability is a non-issue.
perl = creation of in-house tools, advanced scripts
java = business enterprise apps for customers that can be developed fairly quickly
c++ = applications that need less portability and a lot more optimization.
ruby = we had great success using ruby for making great looking reports for customers
Perl may not be exciting or the next thing, but its far from dead. It has a niche that it fills very well.
Re:Why are there still shell scripts anyways? (Score:2)
If you're running a bunch of commands on a system and just verifying return codes, a shell script is the right tool for the job. It's also the right tool for those times when you're working on a very stripped down machine that may or may not have your favorite interpreter installed.
Re:Why are there still shell scripts anyways? (Score:2)
Because I've always got two or three shells open anyway. If I'm using Bash all day anyway, why not write my scripts in it?
There are definitely better scripting languages, but I get a lot more Bash practice than I do Perl or Python practice. That's all there is to it.
Re:Why are there still shell scripts anyways? (Score:2)
Since I use the shell as my preferred work environment, rolling up commands into scripts simply multiplies my existing effectiveness with very little extra effort.
Correspondingly, what I learn from scripting improves my ability to act quickly in a 'live' shell.
Since I work in PHP (mostly), I use it for anything that BASH can't do, since then I have access to my application libraries. If my app was in Ruby I'd use that.
Re:Why are there still shell scripts anyways? (Score:2)
initscripts. If you're not writing them, you aren't a software developer.
Perl server pages. (Score:2)
<<date +%d/%m/%Y>>
works on my pages. ya it prints the date... Powered by PERL.
PS. I can read an write SQL on my web pages. no programing needed.
Re:Why are there still shell scripts anyways? (Score:2)
echo "Lot of the time it's just quick and easy"
Re:Why are there still shell scripts anyways? (Score:2)
print 'I see your point. Perl is just too unwieldy in this case.';
Re:Why are there still shell scripts anyways? (Score:2)
/usr/local/bin/perl: bad interpreter: No such file or directory
Shoot, perl is in /usr/bin and not in /usr/local/bin. So I'll have to ln -s /usr/bin/perl /usr/local/bin/perl and hope it works.
[John]
Re:Why are there still shell scripts anyways? (Score:2)
Try working in an embedded environment. When you have 4 MB of flash and 8 MB of RAM, and python (even a barely usable stripped python) takes 3 MB, you use shell. Most of my programming is on embedded platforms; if I have 32 MB of flash I don't know what to do with it. I use a lot of shell and awk.
Don't even get me started on PERL; it works for some and for others it is the most obtuse incomoprehensible glop you can imagine. Instant cruft. And it takes megabytes of storage.
Just because you can't see it from your house doesn't mean it has ceased to exist.
Why do people thing there's only Linux distros? (Score:4, Interesting)
The script receiving stdout from telnet would watch for input lines, keeping track of what response was expected, and send either a USR1 or a USR2 back to the script outputting commands into telnet's stdin, which would then either bail or send the next command. It was rudimentary pass/fail error checking, but much better and faster than echo a; sleep 1; echo b; sleep 1; echo c...
I often prefer #!/bin/sh as I know it's a common denominator on all *nix systems, and often my scripts work even if the shell is busybox. Yes, perl or python may handle the task more efficiently and with less code, but keeping fresh on Bourne shell makes it easier to fiddle with initramfs images and embedded systems with limited space and no advanced shells. In fact, I'll often stick with it just for the challenge. I like making the original Bourne shell truly sing...
Open your eyes. There's much more to the world than just "every major (and most minor) distros." Even more than just Linux and the BSDs.
Crikey. If nobody learned assembly, we wouldn't even have operating systems.
Re:Why are there still shell scripts anyways? (Score:5, Insightful)
In particular Bourne shell makes gluing other programs together far easier, cleaner, and more reliably then can be done with any of the languages you list...and I say that even given that I'm extremely fluent in at least two of them.
Correctly and reliably handling even a simple "foo | bar | baz" construct in the languages you list can be done, but you're talking about an LOT of non-trivial systems level programming to do it, effectively "coding C inside perl/python/ruby". No, really. Sure, it's very easy to do it wrong and fragile (just call system(), what's so hard right?), but if you actually want to handle it as correctly and reliably as Bourne does out of the box...it's going to take a hell of a lot more code and detailed systems programming knowledge then even most "Sr" Unix sysadmins tend to have. Almost without fail ever attempted I've ever seen in the industry gets it wrong...typically very, very wrong.
Bourne is built to run other programs, to manage the interactions between them, programming logic being the exceptional use. All of the languages you list are built to be self-contained, programming logic being the primary task and communicating with external programs a very secondary use.
Now of course...if your script/program is self-contained and not just wrapping a bunch of exec()s of other programs, then sure Bourne shell is one of the least favorable options.
There's also nothing saying you can't freely mix and match. Most anyone worth their salt does so all the time. To do otherwise would be like saying a web coder could only work in HTML or only work in JavaScript or only work in PHP. We're talking about complimentary languages here, with some overlap.
Use the right tool for the job.
Re:Why are there still shell scripts anyways? (Score:2)
Shell scripting lets you turn things you do on a shell command-line into scripts, without changing them.
That you can then cruft them up with reams of flow-of-control syntax, variables out the wazoo, optargs for days, and trap statements, is just a bonus.
Sure you would have done it better in Python. But if you don't know Python you take what you type into the shell more than once, and make a script out of it.
Re:Why are there still shell scripts anyways? (Score:2)
I still use Bash for most automation tasks because there's no guarantee that the next person in my job is going to be familiar with Perl, Ruby, and Python, but every unix admin is going to know Bash.
If all I need to do is start up a job, make sure it created a non-zero length output file, and log a message if it ends with a non-normal return code, bash is simple and easy and has no real drawbacks.
For anything non-trivial (i.e. a monitoring plugin to do a transaction on our website) than I would use Perl (because it's what I'm most comfortable with, someone else might choose Python or Ruby), but 90% of the time when I need a script to do something, Bash is just fine and I know that the next guy in my job is going to be able to maintain the script.
Re:Why are there still shell scripts anyways? (Score:2)
Why does anyone still use shell scripts anymore?
Because they are the simplest and quickest way to do a lot of things one needs to do when using a computer. Try this in python, ruby, or perl:
the disk is nearly full, find who is the hog
in bash:
du -x / | sort -n
Now do it in another language
Re:Why are there still shell scripts anyways? (Score:2)
It's trivial to do it in perl!
#!/usr/bin/perl
system("du -x / | sort -n");
Re:Why are there still shell scripts anyways? (Score:2)
It's trivial to do it in perl!
#!/usr/bin/perl
system("du -x / | sort -n");
As a career Perl hacker, I sympathise, but you're enthusiasm is misplaced in this situation.
First, system() only gives you the return value of the command you executed, so you haven't got anything useful. Backticks would work, but they're horribly insecure if you're doing variable interpolation.
Second, a more realistic example would look like this:
for DIR in `find /home/ -maxdepth 1 -mtime -7 -type d`; do du -sh ${DIR} | grep '\d+G'; done;
This, too, is easily done in Perl, but why bother when you're just dropping to bash and running various command-line utilities anyway?
For anything that requires data collection and reorganisation, or which implies non-trivial logic (e.g. finding duplicate files across multiple file systems/machines), I agree that Perl is a tremendously powerful tool. Inline POD documentation and input validation are also extremely useful for the long-term health of the system. Most of the glue logic on my production systems is written in it. But as always, it's horses for courses, and for short, simple laps, bash rules.
Re:Why are there still shell scripts anyways? (Score:2)
It's trivial to do it in perl!
#!/usr/bin/perl
system("du -x / | sort -n");
As a career Perl hacker, I sympathise, but you're enthusiasm is misplaced in this situation.
First, system() only gives you the return value of the command you executed, so you haven't got anything useful. Backticks would work, but they're horribly insecure if you're doing variable interpolation.
That's like saying that the original poster's shell command didn't give him anything useful since it just wrote the output to stdout (i.e. his terminal).
My perl script did the exact same thing since all system does is fork() and exec(), so the stdout/stdin are inherited by the child process(es). In this case, since there were shell metacharacters in the command line, my system() argument was actually passed onto the shell for execution, so in that respect you're right that I haven't gotten anything useful - I started a Perl interpreter to call the shell to run my shell command.
The previous poster's suggestion of using exec() was better than using system() since it avoids the overhead of doing a fork first since I don't really care to return to the Perl session.
Re:Why are there still shell scripts anyways? (Score:2)
It's trivial to do it in perl!
#!/usr/bin/perl
system("du -x / | sort -n");
As a career Perl hacker, I sympathise, but you're enthusiasm is misplaced in this situation.
First, system() only gives you the return value of the command you executed, so you haven't got anything useful. Backticks would work, but they're horribly insecure if you're doing variable interpolation.
That's like saying that the original poster's shell command didn't give him anything useful since it just wrote the output to stdout (i.e. his terminal).
But in shell that's exactly what you need for the next step of processing. In Perl you'd want to capture this in a variable, or split it on newlines to produce an array value, or something. That's... what, backticks? I am no Perl master.
Re:Why are there still shell scripts anyways? (Score:2)
But in shell that's exactly what you need for the next step of processing. In Perl you'd want to capture this in a variable, or split it on newlines to produce an array value, or something. That's... what, backticks? I am no Perl master.
He was debugging a disk space problem and wanted to see what was using the disk space. No need to put it into variables, no need to build an array, he wanted to see where his disk space was going. Sometimes even in perl, you don't want to do any complicated data manipulations, you just want to see the output.
Though admittedly, there'd be no reason to run his bash commands in Perl, but hey, he asked, so I provided.
He said this:
in bash:
du -x / | sort -n
Now do it in another language
Nothing about storing it in variables, nothing about building an in-memory data structure to calculate average disk usage, all he wanted to do was see the top folders in disk usage.
spellening (Score:2)
It's trivial to do it in perl!
#!/usr/bin/perl
system("du -x / | sort -n");
As a career Perl hacker, I sympathise, but you're enthusiasm is misplaced in this situation.
"you're" is a contraction. What do you suppose it's a contraction of?
OK, 5 seconds and here it is in perl (Score:2)
You left yourself wide open. Here's my answer:
#!/usr/bin/perl
exec("du -x | sort -n");
Would I use it in place of
#!/bin/sh
du -x | sort -n
No. But it's not something I have to "try" very hard to do in the same number of lines and not many more characters.
Re:Why are there still shell scripts anyways? (Score:2)
You listed the options backwards. Most UNIXes (you might be shocked to find that Linux isn't the only Unix variant people use) do not ship with Python, and I do everything I can to avoid having Ruby installed on even my Linux systems. And while I love perl, you are the first person I've heard describe its syntax as sane. :)
Re:Why are there still shell scripts anyways? (Score:2)
Why does anyone still use shell scripts anymore?
Because I'm a Unix Sysadmin?
Shell scripting is something I haven't felt the need to do in 5 years.
I'm guessing you're not a Unix Sysadmin any more?
If we were interviewing someone who said they hadn't written a shell script in 5 years, I'd have to spend some extra time querying the reasons and then think twice about hiring him. See, we use shell scripts. For init scripts, for simple tasks, for data gathering; loop through 300 systems getting the uptime for instance. Someone we hire would need to be able to perform maintenance on the scripts and hopefully not break an existing script or start using something like Python that the other sysadmins may not know and have to either learn in order to maintain your scripts or convert to regular shell scripts again.
See I've dealt with people who have their own little way of doing things. We had one sysadmin who felt Rexx was a great scripting language. Which was fine for him but when he left, we had to convert his scripts into a shell script in order to continue to maintain things.
So we have a rule that system scripts are done using a specific template, which is editable of course in case of need, which manages several aspects of system scripts including checking for multiple copies of the script and killing sessions if scp to a system hangs for some reason.
[John]
Re:Why are there still shell scripts anyways? (Score:2)
Shell code is much easier to write then any other language, thats why we type it into our xterms after all instead of Python code. So naturally shell one-liners grow into shell scripts. There is also a mismatch between what Python/Ruby provides and what your shell provides, thus converting shell scripts into Python/Ruby isn't exactly straight forward and its easier to keep things as shell instead of doing a complete rewrite.
Re:Why are there still shell scripts anyways? (Score:2)
Why does anyone still use shell scripts anymore? Every major(and most minor) distro ships with python, ruby, and perl either built in or a trivial statement away. These languages give you access to basic OS functions and much more sane syntaxes. Shell scripting is something I haven't felt the need to do in 5 years.
Then your tasks are different from mine. There are things that can be done much better in a shell script than in Perl or Python. Anything that executes a lot of other programs for example. Anything that pipes them together. I use Perl and/or Python a lot for many things, but shell scripts still have their place. (And if you choose to use bash rather than /bin/sh the language isn't *that* primitive.)
Re:Why are there still shell scripts anyways? (Score:3)
Maybe because bash is installed by default on most Linux/UNIX systems and is portable across all yoru installed base.
perhaps they exist but in 20 years I've not see a Linux distro without Perl.
Or maybe your security requirements disallow python, or ruby or whatever to be installed.
if they disallow perl then shell needs to be disallowed too
Or that running shell scripts using many built-ins run faster than the interpreted languages?
this is patently false. Perl is vastly faster than shell even taking into account load times when you do anything significant. Bash scripts that do anthing important generally are calling things like awk and grep and sed to parse things and these invocations are dramatically slower in a loop than a single invocation of perl. Additionally, perl I/O is in many case faster than standard unix operations that move large blocks of data.
finally invoking and disposing a command like awk many times in a row makes for very poor memory management and system resource utilization compared to keeping one program like perl resident.
So don't try to argue this on speed or system resource utilization.
Bash scripting features seem to stay more stable over time vs. other scripting languages that are constantly changing.
huh? perl is very static and backward compatible.
moreover, it's the helped commands that bash needs like grep and awk and so many other system resource introspection calls that one cannot count on being present or taking exactly the ame arguments.
even a braindead command like "cp" is not only different on Linux and BSD but it's command arguments have changed over time. Whereas is perl the commands needed to copy files have not changed in 20 years.
The app teams within your company may want to constantly upgrade the installed version that might break you scripts/programs.
the only time i've seen broken perl was the 5.8 to 5.10 switch changed the timing of a few things.
And bash is dirt simple to learn and implement for non-programmers.
Perl is very very very close to bash in syntax
Re:Why are there still shell scripts anyways? (Score:2)
finally invoking and disposing a command like awk many times in a row makes for very poor memory management and system resource utilization compared to keeping one program like perl resident.
Poor memory management? The awk binary and its libraries will stay resident in the pagecache between invocations, and awk+bash has a lower memory footprint than perl (on my system, running perl uses around 19MB of Virtual memory, awk+bash is around 14MB).
So, it could be argued that running awk+bash is better for memory overhead.
fork+exec overhead is so low that unless the script is calling awk thousands of times, it's not worth the time to even think about it. It takes around 2ms for me to call awk from a script.
Re:Why are there still shell scripts anyways? (Score:2)
I agree with you, right up until you claim perl is remotely close to bourne syntax...
Sure, some of the latest bash' isms look perl' ish, but that's about it. Perl gets -1 trillion points for the very existence of the "unless" function... You could read through an entire program, line by line, looking for any possible issues, only to find an unless statement that is always false... Meaning none of that is ever executed anyhow.
Perl is notorious for how horrendously unreadable and un maintainable it is. Bourne has nothing of the sort to contend with.
Nothing's perfect, but if it's not massive, and not performance critical, I'm writing it in bourne.
Re:Why are there still shell scripts anyways? (Score:2)
Maybe because bash is installed by default on most Linux/UNIX systems and is portable across all yoru installed base.
perhaps they exist but in 20 years I've not see a Linux distro without Perl.
I've seen many. Usually they are embedded systems, where memory is at a premium.
Or maybe your security requirements disallow python, or ruby or whatever to be installed.
if they disallow perl then shell needs to be disallowed too
Nonsense. From a security point of view there's a lot of sense in minimising the amount of code on a system. If you have bash and perl running, and someone discovers a vulnerability in perl, you're wide open. If you just have bash, you're not. Of course, the same applies the other way round too - but there is a strong security argument for having "less stuff" installed and running wherever possible.
Or that running shell scripts using many built-ins run faster than the interpreted languages?
this is patently false. Perl is vastly faster than shell even taking into account load times when you do anything significant. Bash scripts that do anthing important generally are calling things like awk and grep and sed to parse things and these invocations are dramatically slower in a loop than a single invocation of perl. Additionally, perl I/O is in many case faster than standard unix operations that move large blocks of data.
finally invoking and disposing a command like awk many times in a row makes for very poor memory management and system resource utilization compared to keeping one program like perl resident.
So don't try to argue this on speed or system resource utilization.
You're right that it's not a hands-down win for bash, not by a long way. But there are tradeoffs - for instance, in embedded systems (once again), the footprint of bash alone is much smaller than that of bash+perl.
Bash scripting features seem to stay more stable over time vs. other scripting languages that are constantly changing.
huh? perl is very static and backward compatible. moreover, it's the helped commands that bash needs like grep and awk and so many other system resource introspection calls that one cannot count on being present or taking exactly the ame arguments.
even a braindead command like "cp" is not only different on Linux and BSD but it's command arguments have changed over time. Whereas is perl the commands needed to copy files have not changed in 20 years.
You do have a point here - but then, copying a file in perl is rather more work than it is in bash (especially once you take into account error checking, recursive copying etc...)
The app teams within your company may want to constantly upgrade the installed version that might break you scripts/programs.
the only time i've seen broken perl was the 5.8 to 5.10 switch changed the timing of a few things.
Were you not around for the perl4->5 change? That broke scripts... though to be fair I don't really see this as an argument either way, because bash (and the utilities called from it) change from time to time too...
And bash is dirt simple to learn and implement for non-programmers.
Perl is very very very close to bash in syntax
I actually disagree that bash is very simple to learn for non-programmers. It's very easy to write simple scripts that appear to work, but break if they're passed a parameter containing a space or an asterisk. In this respect perl is better - it is (by design) much less vulnerable to special characters in input. But I also disagree that the syntax is very close to bash - superficially there are a lot of dollar signs in both, and the quoting *looks* similar, but that's about as far as it goes...
Re:Why are there still shell scripts anyways? (Score:2)
When I need to rescue a server, I boot with a recovery USB flashdrive, and then I have a full linux environment at my disposal Including Perl (and probably Python and Ruby, but I've never checked).
Gone are the days when you only get what's in /sbin and if you can't fix it with cat and sed then you're out of luck.
Re:Why are there still shell scripts anyways? (Score:2)
Try again... I'm not taking a flight to the data center to plug in a usb flash drive.
Re:Why are there still shell scripts anyways? (Score:2)
Or you could use perl and `` to run whatever you want.
I use bash all the time, but sometimes perl is the way to go.
Useful (Score:2)
Re:Useful (Score:5, Informative)
Re:Useful (Score:2)
reviewer isn't a sysadmin (Score:5, Insightful)
Seriously? So you write your application's
Re:reviewer isn't a sysadmin (Score:2)
csh if on solaris
Speaking of csh on solaris, I had a devil of a time figuring out why my csh scripts never worked until I realized that I needed to always put a newline character on the last line (the system wouldn't parse the last line without it). I slowly learned to hate csh's oddities as I learned tcsh (and later, [ba]sh).
Re:reviewer isn't a sysadmin (Score:3)
csh on Solaris? All scripts are generally in /bin/sh on my Solaris systems.
[John]
i can forkgive if you can forkget (Score:3)
So the bash fork bomb got popular back while I was at UNH. If you're not familiar with it, it's similar to this: ;(){ :|:& };: (I changed one character so that you dont paste it on recommendation) The way it works is trivial, and it's mystique is only in that it uses punctuation marks in lieu of letters for its own name: make a new function called :, run it and force the output into a new fork of itself in the background; then a final call to the new function. h4rdc0r3 1337.
So, we had this UNH policy that if you hung a shared unix dev box, you got evicted from CEPS (the college for the engineering/cs kids). They could give you a warning or go straight to evict. Their call.
I wrote to their head unix admin once, and with some humility, cause I knew they knew this, pointed out that they could just change the system's ulimit values to disallow casual fork bombing. The default system ulimits on the distro back then (still?) allowed a single user to consume all memory. So we saw a couple kids run the fork bomb and they never actually got evicted, cause maybe 40 grand tuition outvotes a cranky unix admin's wanton lust for cruelty.
Anyways, as a student, my issue was this - set the damned ulimit so that my editor session doesn't get wiped with my homework -- you can't argue for frequent backups when more than 0 seconds of hard work are arbitrarily on the line solved by a simple config change.
But the UNH unix admin guy replied to me and told me that yeah, he knew about ulimit back when I was in diapers, but he wasn't going to change it up. And a few more times that semester, I lost a couple lines of uncommitted code due to some clown pressing enter on a dare.
Fork ya later,
Comment removed (Score:5, Insightful)
Re:Obviously, these guys haven't used UNIX. (Score:2)
There's a reason we use /bin/sh (Bourne) to write scripts, and it's the same reason we use vi to edit them - mainly it's available on ALL UNIX systems. Despite widespread adoption, bash is not universally available - sorta like EMACS.
If I stick it in root's cron on, say, an AIX system and it pukes I'm not interested in rewriting somebody's bash script to handle running under sh, csh or ksh. I want it written by its original author in sh. And I don't want to hear about a shebang line (#! /bin/bash) - if I haven't installed bash, it'll still puke.
That makes good sense if you are writing scripts that you will have the need to run from maintenance mode, but that's a pretty small amount of administrative scripting when it comes down to that.
However, if you are writing all of your shell scripts with the assumption that you will not be able to access something as widely available on all UNIX platforms as bash, even when those scripts are not needed in maintenance mode and/or make reference to things that wouldn't be available in maintenance mode anyway, that's just masochistic.
Re:Obviously, these guys haven't used UNIX. (Score:2)
> mainly it's available on ALL UNIX systems
Mainly it doesn't matter since this book is about LINUX shell scripting.
Besides Bash, a lot of tricks and utilities are only available if you assume a Linux environment.
Re:Obviously, these guys haven't used UNIX. (Score:2)
There's a reason we use /bin/sh (Bourne) to write scripts, and it's the same reason we use vi to edit them - mainly it's available on ALL UNIX systems. Despite widespread adoption, bash is not universally available - sorta like EMACS.
But not all GNU/Linux systems. Gentoo is a prime example. Nano instead of vi. And as sibling stated: almost no GNU/Linux system has true sh, they symlink sh to bash.
Oh, and don't stick things in a crontab on Mac OS X systems. Mac OS X prefers XML plist files for scheduling and startup.
Re:Obviously, these guys haven't used UNIX. (Score:2)
You, sir, understand the Unix philosophy.
Re:Obviously, these guys haven't used UNIX. (Score:2)
Not all scripts are meant to be re-usable.
I often find myself writing a script directly in my terminal, without going through a text editor. A simple for i in `find the_files_i_want_to_do_something_with | preprocess_the_input`; do what_I_want_to_do_on_those_files $i; done.
In this particular, I'm just using the shell I happen to be running (bash). I don't have to care about compatibility. Whatever works to get my job done is enough.
Actually, whenever I want to write a re-usable script, I write it in Python.
Re:Obviously, these guys haven't used UNIX. (Score:2)
A longstanding pet peeve of mine are systems that symlink /bin/bash to /bin/sh
Doesn't bash emulate sh if it's called as sh?
Re:Obviously, these guys haven't used UNIX. (Score:2)
Perhaps but if you're in single user mode and can't log in because the shared library isn't available, what do you do now?
At least that's where my pet peeve would come from. It doesn't happen often but when it does, it's a pain to get the system working.
[John]
Re:Obviously, these guys haven't used UNIX. (Score:2)
Re:Obviously, these guys haven't used UNIX. (Score:3)
bash uses shared libraries and sh doesn't. If you lose the /lib directory (or /usr/lib, or whatever), you lose the ability to log in if a system crashes to single user mode and the library file system didn't mount. Then it's boot to CD to recover which is great fun on a remote system (but not always impossible with virtual mounts, etc).
[John]
Re:Obviously, these guys haven't used UNIX. (Score:2)
This link has a tabular description of the differences between sh & a small raft of other shells [faqs.org], including bash. Bash syntax is very similar similar to sh, but there are a few gotchas, which I'm too long out of the game to remember.
This book gets the details critically wrong. (Score:4, Informative)
I've just had a look through the book via Amazon's Look Inside.
I simply cannot fathom how so many authors manage to get a book published on a subject they don't even master properly. "Shell scripting" is by far the worst in this category. So much that I recommend anyone to run screaming in the other direction when you see mention of "shell scripting" in any title. If it mentions "Linux", it's usually also an excellent indicator of junk. Not to take away from the beauty of the Linux kernel (which obviously has nothing at all to do with shell scripting, but let's ignore that fact).
I obviously haven't read the book, but when I skim through it and I see failure to quote parameters, failure to recognize the difference between executing a script with /bin/bash in the hashbang and running it by passing it as an argument to sh, and more, I know for a fact that I'm going to be sitting in #bash trying to re-educate poor misguided souls for as long as I can suffer it.
To those interested in the subject: Do not read this book. Do not read this "Advanced Bash Scripting Guide" mentioned in this review. You will end up writing junk.
Go see Greg Wooledge's wiki, it's got an extensive FAQ of actually useful issues (http://mywiki.wooledge.org/BashFAQ), it's got a great newbie guide (http://mywiki.wooledge.org/BashGuide), it's got a great cheat sheet (http://mywiki.wooledge.org/BashSheet), and much more. Best of all: Unlike that broken ABS, they're all community supported and unlike this book, the knowledge is free.
Re:This book gets the details critically wrong. (Score:2)
What's wrong with the Advanced Bash Scripting Guide at tldp.org?
It's not perfect, but it will get you along way.
Re:This book gets the details critically wrong. (Score:2)
The main issue I have is that the "Advanced Bash Scripting Guide" seems to be fascinated with listing every possible way to do the things it covers in bash. But missing from that tutorial is any notion of which of those approaches is the best one in any context. Certain shell constructs are best avoided, in favor of ones that are either more portable to other shells, more resilient in the face of bad data, or easier to maintain. But you get no sense of this from the guide; just a set of ways to hack the low-level code into place.
Re:This book gets the details critically wrong. (Score:2)
I'm of the opinion (just an opinion, mind you) that shell scripting is a bit of a black art and very quirky and unpredictable. I do my shell scripting by "hack and test, repeat."
I think it's very dangerous when people claim that shell scripts can be portable (some people abhor bash and say that you should stick with sh) especially since they rely on the commands and utilities installed on any particular system. Are they GNU, BSD, SysV, POSIX? What versions are they? What directories are they in etc.
I think that shell scripting to any extent beyond 20 lines, say, is way too complicated for beginners. People should be introduced to programming through a sensible and simple high-level language before they are condemned to shell scripting.
Shell scripting can be fun in a perverse kind of way if you enjoy all the "clever" and dirty hacks that can be done, but, as you say, there are may pitfalls for the beginner.
Re:This book gets the details critically wrong. (Score:5, Funny)
Go see Greg Wooledge's wiki
Just not all at once, please.
Thank you! (Score:2)
I had been limping long with the Advanced Bash Scripting Guide and Google searches for a while. The Wooledge guide looks like a vast improvement. Thanks.
Re:This book gets the details critically wrong. (Score:2)
This might be the appropriate place to mention the venerable Unix Power Tools [oreilly.com].
It goes through most of the common Unix commands (including Bash, but also sort, uniq, awk, sed, etc.). It teaches you about redirection and piping, customizing your shell prompt, xterms, X, remote X, Unix file times and permissions, find/exec, diff, head/tail, grep, vi, emacs, jobs, cron, time, regexes, sed, bash scripting, tar, rcs/cvs, perl/python, /dev, symlinks, tty's, ssh, users/groups.
In short, everything you ought to know before opening a root shell.
It's written in a funny/witty (for geeks) manner.
yahoo (Score:2)
The only interesting thing about this book was to find the author's web site and see how many hoops there are nowadays to get through the Yahoo interview process. I've a friend who managed to get a senior sysadmin role at Yahoo a few years ago who can barely program and has no computer science knowledge whatever - his only previous experience was some ISP admin work, although he was a tremendous suck-up to^W^W^Wgood friends with a few core FreeBSD developers. Anyway, I had the impression that the company was a has-been which runs on inertia.
Boy Goatsex is out in force today... (Score:3)
Every topic is littered with them...
Re:Thanks (Score:2)
Re:Who uses shell scripts today? (Score:2, Interesting)
Re:Who uses shell scripts today? (Score:2)
It really should just expand them when the comment is submitted.
Re:Who uses shell scripts today? (Score:2)
The problem is then these folks will link to blog pages that they setup to host this material. Some have already started doing that.
Re:More efficient scripts (Score:2)
Agreed!
This book is probably interesting, but in my opinion if you are doing anything beyond a basic set of commands run in order (with _maybe_ some very primitive control statements) .. you are far better off using a more powerful scripting language... or even a real programming language.
Personally I don't like shell scripts at all for anything more complex than a temporary convinience. From what I've seen they are usually poorly written (lacking proper error handling and logging), are usually riddled with dependancies, and fragile. Oh, they added a few spaces in the output of some command you call? And your script didn't notice and just kept right on doing? Well now you're screwed!
Obviously you can have the same kind of problems with a real programming language.. but shell scripts just seem to encourage bad things to happen.
Re:More efficient scripts (Score:2)
Is this what you were thinking of?
ls -l | awk {'print $5'} | sort | tail -n1
Re:More efficient scripts (Score:2)
ls -l |sort -k3|tail -1
Re:More efficient scripts (Score:2)
Two processes.
Comment removed (Score:2)
Re:Remember the torture (Score:2)
That's somewhat clever, but some of us do know what base-64 encoding is.