Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×

Practical File System Design with the Be File System 258

Posted by timothy from the online-knowledge dept.
erikharrison writes "Dominic Giampaolo's Practical File System Design with the Be Filesystem has been around since 1999 - not exactly a new book. The book has been out of print for a time now, however, so Dominic made the book available in PDF form on his website. With this public release of the book, and the BeOS rising to join the ranks of OSs that won't die (hi Amiga!) it makes sense to take a look at what the book has to offer us today." Read on for the rest of Harrison's review below to see just what that is -- it covers a surprisingly broad range.
Practical File System Design with the Be File System
author Dominic Giampaolo
pages 227 pp
publisher MORGAN KAUFMANN PUBLISHERS, INC.
rating 8.5 of 10
reviewer erikharrison
ISBN 1558604979
summary Discusses implemeting a file system, using the Be file system as example

Table of Contents

  • Chapter 1 Introduction to the BeOS and BFS
  • Chapter 2 What Is a File System?
  • Chapter 3 Other File Systems
  • Chapter 4 The Data Structures of BFS
  • Chapter 5 Attributes, Indexing, and Queries
  • Chapter 6 Allocation Policies
  • Chapter 7 Journaling
  • Chapter 8 The Disk Block Cache
  • Chapter 9 File System Performance
  • Chapter 10 The Vnode Layer
  • Chapter 11 User-Level API
  • Chapter 12 Testing
  • Appendix A File System Construction Kit

First thing to note is that Giampaolo is not a great writer, nor is he a bad one. He does not have the gift that some tech writers have of making both an interesting technical document and a fun read. His style is very straightforward - introduce idea, explicate idea, summarize idea. On the other hand, he knows his topic inside and out, and has an obvious enthusiasm for the material, and a real talent for saying things simply without dumbing it down, and his occasional dry wit makes the book a surprisingly easy read.

Giampaolo is doing two things - discussing designing filesystems in general and documenting the Be filesystem. He does both well. BeFS has some advanced features - arbitrary metadata, attribute queries, and indexing. The desire to support these features influences the overall design of the system, but Giampaolo shows how changes to that design change implementation details. The result is a good overview of how a file system works, the trade-offs in optimizing for a particular usage pattern, and how to design one yourself.

The book can be roughly divided into three sections: the first is an overview of how filesystems work and some of the concepts that you encounter - extents, inodes, B-trees, superblocks, and the other standard pieces of a filesystem. Included in this early section is a good high-level overview of the design of five other file systems: BSD FFS, Linux's ext2, Macintosh HFS, Irix XFS, and Windows NT's NTFS. The coverage here strikes a proper balance between too much and too little information. Giampaolo prefers to show rather than to tell, and these filesystem overviews make the connection between design, performance, and features perfectly clear, and provide a solid background to talk about a specific implementation in detail - namely BeFS.

The second section is the bulk of the book - how to implement a filesystem from the ground up, leaning heavily on the BeFS implementation for examples. This is the most straightforward part of the book. Giampaolo covers a single issue in design and implementation in a "Here's the problem, here's and overview of possible solutions and their drawbacks, here's how I did it, now lets summarize" manner. Again, Giampaolo's style makes this an easy if somewhat dry read. As a filesystem and kernel ignoramus, I would have appreciated a slightly more detailed coverage of how all of the various data structures get to disk - how are they serialized, whether endianess is an issue, etc. The BeOS was pretty portable, running at one time or another on the AT&T Hobbit processor, PowerPC, and x86 - I would have liked to have seen portability issues discussed, however, BeFS wasn't written until after the move from the Hobbit to PowerPC, and the book was written prior to the move to x86, so the lack of coverage is reasonable.

Even considering the plain Jane style of this middle section, there are a few gems. The coverage of journaling is excellent, and while I've long understood journaling from a 10,000 foot perspective, this really made me understand the underlying concepts, combined with simple code snippets that helped understand implementation. The Allocation Policies chapter showed in clear terms that disk access is a major bottleneck, and filesystems have become very sophisticated in their optimizations.

The third section of the book deals with some of the more indirect concerns in implementing a file system; specifically, interacting with the kernel, designing a user level API and the major role of testing in filesystem development. This is the one place Giampaolo's writing shines. He really is a good teacher, and this section affords him the chance to talk about the broader perspective of OS design, and even recount a few war stories. For example, in terms of parentage, the BeOS has BSD and classic MacOS as its father and mother. In a few places, such as the Storage Kit API covered in chapter 11, this heritage shows some signs of less-than-seamless integration, and this offers Giampalo a chance to wax philosophical on the nature of OS design, company politics, and the pressure of shipping dates.

In short, the book lives up to it's title. The author is a pragmatist, and offers a clear roadmap for those who have a need to work with low level filesystem implementation. His emphasis on testing, careful optimization, and data structure protection not only helps to show the pitfalls of filesystem work, but also offers a Swiss army knife of techniques to dodge them. The book concludes with a short appendix which covers a file system construction kit, allowing a would-be implementor to begin work on his own filesystem safely without worrying about killing his hard disk. All in all, a solid read.

Here's a link to Practical File System Design with the Be File System as a PDF; you can also look for a used copy at Barnes & Noble. Slashdot welcomes readers' book reviews -- to see your own review here, carefully read the book review guidelines, then visit the submission page.
This discussion has been archived. No new comments can be posted.

Practical File System Design with the Be File System More

Practical File System Design with the Be File System

Comments Filter:

  • Mirror with PDF (Score:5, Informative)

    by BiggestPOS ( 139071 ) on Monday May 10, 2004 @03:46PM (#9110496) Homepage
    http://biggestpos.com/pfs/ It took me so long to download the file I thought I'd put it on a faster server for you guys.
  • One more backwards-looking text... it simply ignores that the future is relational databases as a filesystem.

    • Re:Future is relational databases (Score:4, Insightful)

      by happyfrogcow ( 708359 ) on Monday May 10, 2004 @03:54PM (#9110574)
      and just where are you going to store that database? on another database acting as a filesystem? or will you have a db built into the OS, right next to your flight simulator, and supercolliding monkey accelerator?

    • You are right, but not only relational...the future is object-oriented databases.
      • You are right, but not only relational...the future is object-oriented databases. ... implemented in XML.
    • Maybe but maybe not. There are some performance issues to look at not to mention that even if the relational database uses a raw partition it is still use some sort of filesystem to store its data on disk and to manage free space. Even if it is one of it's own design. An how do you know ti will be a realtional database? Maybe it will be an OO database.

      • >

        maybe not

        Only if inertia and ignorance continue to hinder us.

        >

        here are some performance issues

        No, this is a physical issue. The relational says nothing about the physical level, thus leaving the implementor total freedom to achieve the best performance possible. This is typically better than what is possible navigationally.

        >

        even if the relational database uses a raw partition it is still use some sort of filesystem

        Not at all. What makes you think so?

        >

        how do you know ti wil

        • The main issue here is not if this is possible but is it going to happen.
          The amount of just I am sure I am right and all others are wrong is mind numbing. I can see how a file system based on a relationl database could work. I am just not sure that it is the ultimate solution.

          As to it still requireing a file system. Yes even if they database uses a raw partiton it will need to keep track of free and used blocks. To me this is a filesystem.

          As to OO databases already being considered useless. So far you ar

          • >

            is it going to happen

            Not soon. There are currently several problems:

            First, OS research is stagnated. It tends to focus on low-level stuff like distribution or microkernels, or GUI stuff like Sun's. Anything else just doesn't happen, because it is considered that the POSIX model can't be overthrown, be it in its implementations like the GNU system or in its corruptions like MS Windows.

            That said, there is the GNU Hurd. I have a suspicion it is meant by Stallman to be a stepping stone from POS

            • I have to agree that many parts of CS is not just stagnate but pretty dull.
              OSs, ISA, and GUIs are pretty much stagnate.
              I was pondering this relational database as a file system. But I was wondering how do you map files and directories to tables and tuples?

              Yea Hurd is... Well not moving fast. I know Unix is good but is it the best that we can do? I wonder if something like Plan9 or openmosix is an interesting direction to look in. Lets face it networks of computers is the dominate system and is likeily to g

      • There are some performance issues to look at not to mention that even if the relational database uses a raw partition it is still use some sort of filesystem to store its data on disk and to manage free space.

        OK, IANAFE (filesystem expert).

        It wouldn't need a full-blown filesystem. That "filesystem" would be extremely fast and simple for storing and retrieving data. The database itself would be the real filesystem in terms of abstraction.

        From the book:

        The two fundamental concepts of any file system are

    • Re:Future is relational databases (Score:5, Interesting)

      by ceswiedler ( 165311 ) * <chris@swiedler.org> on Monday May 10, 2004 @04:02PM (#9110659)
      It's interesting that one of the features of BeFS is its metadata indexing capabilities (which are the beginnings of a relational model).

      Clearly the BeOS designers agree with you.

      It's also interesting that the author spends quite a while discussing how difficult it is to do well (particularly performance-wise) and how they almost left it out (IIRC) and/or had to limit its scope.

      Clearly the BeOS developers think you're wrong.

      And personally, I'll believe people who have actually tried to implement the technology in question over people who say others should do so.

      • >

        its metadata indexing capabilities (which are the beginnings of a relational model)

        No, they could be the beginnings of an implementation of the relational model.

        >

        the BeOS designers agree with you

        If they did they would have gone full ahead instead of taking half measures.

        >

        It's also interesting that the author spends quite a while discussing how difficult it is to do well (particularly performance-wise) and how they almost left it out (IIRC) and/or had to limit its scope.

        No one ever

        • Re:Future is relational databases (Score:5, Informative)

          by no reason to be here ( 218628 ) on Monday May 10, 2004 @04:27PM (#9110881) Homepage
          Actually, Be started with a full database instead of a file-system. They found it to be incredibly slow and crash-prone, and so they developped the marvel that is the Be file system.
    • Maybe you're trying to be funny, but BeFS was renowned for its database-like properties that allowed for easier and quicker searches to find data.
      • >
        Maybe you're trying to be funny

        Am not.

        >
        BeFS was renowned for its database-like properties

        That's not what we need. What we need is a full relational database as the core of the OS, with a relational-enabled language as the primary systems programming language.

    • Well. I'll bite.

      The text isn't "backwards looking". It is a discussion of how one man implented a file system, and how filesystems are generally implemented. Dominic himself comments that his book is a textbook of sorts (or the textbook he wishes he's had in Graduate School, where he did FS work) and that BeFS is an attempt to implement already existing functionality of the BeOS (where they were using a true database as a file system) using tried and true methods. In fact he bemoans that he couldn't do any

      • >

        It is a discussion of how one man implented a file system

        Without taking in account 30 years of research and development of the relational model.

        I can understand it may sound strange, but it has been that long since hierarchies became obsolete as a way of organising -- not necessarily of presenting -- data.

        >

        the relational model is a great way to organize your data. You want to know how x relates to y in your data set

        No, that's a relationship, not a relation. The relational model isn't abo

    • i honestly don't see why the future is so written in stone as being "relational databases."

      BeFS has all of the capability that a relational database has in a hierarchical arrangement with POSIX properties.

      i don't see any need for anything more powerful than that.

      what is it about relational databases that makes it so critically important? why isn't BeFS good enough for you? (as a side note, have you read the book?)

      • >

        why the future is so written in stone as being "relational databases."

        Because the relational model is the only real data model in existence, giving us the power and performance we need by being simple, expressive, and by separating the logical from the physical level.

        >

        BeFS has all of the capability that a relational database has in a hierarchical arrangement with POSIX properties.

        That is part the problem. More powerful views of files should be possible.

        >

        have you read the book?

        No.

    • Why would you want an relational database for your file system. maybe it's just me but I remember everypoint of every piece of data. My mind may use a relational system to keep everything in place, but my phyiscal world is seperated into specific locations for each object.

      When I go for a file, a DVD or one of the swords I know where it will be and what it will contain. Everything has a place and everything in it's place.

      If you pull all that apart, and interconnect it then it doesn't have it's place any

      • >

        maybe it's just me but I remember everypoint of every piece of data

        It is just you. Or you're not quite truthful.

        >

        If you pull all that apart, and interconnect it then it doesn't have it's place anymore

        There is no interconnection. In fact, the relational model does away with pointers (and relationships) and substitutes relations instead.

        The point of the relational model is not doing away with hierarchies as ways of presenting and organising data at the user or physical levels, but as ways

  • It's a nice thought. (Score:5, Interesting)

    by DiscordOfFive ( 778099 ) on Monday May 10, 2004 @03:49PM (#9110535) Journal
    I generally think it's a good thing when books get put up (legally) on the web like this. My personal view on BeOS is that it's more of a hobby OS than a production one, but a book that details the workings of a complex system is useful. After all, maybe it'll help form the basis of a new, advanced FS.

    • The reason he is at Apple is that he was brought in because of his work on the BeOS FS, so a lot of the ideas and inner workings (AFAIK) are being put into OSX.

      I would hunt some links out but I can't be arsed, but El Reg did a very good interview with Dominic a couple of years ago that was quite a good read.

  • Without being too cynical (Score:5, Insightful)

    by Timesprout ( 579035 ) on Monday May 10, 2004 @03:50PM (#9110543)
    He does not have the gift that some tech writers have of making both an interesting technical document and a fun read.

    But for most people this is seriously dull subject matter. Oscar Wilde would stuggle to get a chuckle out of this stuff.

  • Mirrors: (Score:3, Insightful)

    by Anonymous Coward on Monday May 10, 2004 @03:51PM (#9110552)
    Mirrors of the PDF:

    http:/beos.spb.ru/program/105/practical-file-sys te m-design.pdf.zip

    http://users.aber.ac.uk/mmb9/data/practical-file -s ystem-design.pdf

    http://www.funtech.org/downloads/Temp/practical- fi le-system-design.pdf

  • Silly submitter... (Score:5, Funny)

    by grub ( 11606 ) <slashdot@grub.net> on Monday May 10, 2004 @03:54PM (#9110572) Homepage Journal

    the BeOS rising to join the ranks of OSs that won't die

    An OS isn't dead or dying until Netcraft confirms it.
    • and since Be lives on [yellowtab.com], it will probably be awhile before netcraft confirms it. At least Yellowtab, is releasing something whereas amiga hasn't released anything tangible (although they say they have) since os v4.
    • Actually, Netcraft only confirms what is already present in the most reliable data measure we have about the life of an OS: The number of posts about it to usenet. So always check usenet postings before shouting "Red ink! River of blood!"

      Also, it would help to be aware of how long it takes to copy a 17 M file from one disk to another under the OS.

  • Just what I always wanted (Score:5, Funny)

    by Anonymous Coward on Monday May 10, 2004 @03:54PM (#9110573)
    A practical file system with an impractical OS!

  • Holy Crap. I'm a Be-diot. (Score:4, Funny)

    by torpor ( 458 ) <ibisumNO@SPAMgmail.com> on Monday May 10, 2004 @03:57PM (#9110608) Homepage Journal
    All those years I've slagged off Amiga blow^H^H^H^Hdie-hards, and here am I with a rev-a. BeBox I refuse to part with. I love the damn thing. I never noticed it, but I've got Amigazoids' Disease.

    Why, oh why can't someone invent a "Parallel Universe Chunnel" so I can get myself a laptop BeBox. tiBooks come close, but Apple crack is still crack!

    Ah well. At least I have something else (the .PDF) to put on my Amig^H^H^H^HBeBox, beautiful and blue though it sits in the corner, gathering dust, as if there's nothing else to do with it ... ;)

  • ironic fact (Score:5, Informative)

    by Power Everywhere ( 778645 ) on Monday May 10, 2004 @03:58PM (#9110620) Homepage
    Apple considered buying Be a few years ago for BeOS and opted for NeXT instead. Now, years, later, they have hired several Be engineers to work on the Mac OS X filesystem. It looks like Apple is getting Be without having to buy the whole company. Be fans, look at Tiger as an upgrade for your favorite OS.

    • Re:ironic fact (Score:3, Interesting)

      by nacturation ( 646836 )
      ...especially when you consider that both NeXT and Be were started by former Apple VPs (Steve Jobs and Jean-Louis Gassee [macobserver.com]).

    • Re:ironic fact (Score:2, Interesting)

      by ckelly5 ( 688986 )
      Yes, I loved BeOS back in its heyday. When I learned that Giampaolo had gone to Apple, I figured he'd be beefing up the FS. His first work was probably the Journaling FS that is prevalent in Panther, and the rumored metadata in Tiger has his name all over it...
  • I lent a copy to a friend a while ago. I just asked for it back.

    When's the last time you had a PDF book as bathroom reading?

  • BitTorrent (Score:5, Informative)

    by gspr ( 602968 ) on Monday May 10, 2004 @04:05PM (#9110691)
    It may be redundant - if it is, just moderate it as such, but here is a Torrent [dyndns.org], so that we don't completely destroy the nice BiggestPOS' mirror.
    I'll seed it for an hour or so.

  • Interesting (Score:3, Insightful)

    by 0xC0FFEE ( 763100 ) on Monday May 10, 2004 @04:07PM (#9110710)
    This seems like quite an interesting read. But what I wonder is if BeFS is encumbered by any patent... If so, who own them now, do they plan on enforcing it, etc. Would be oh so cool to have an open source BeFS implementation.

  • Other possibility (Score:3, Informative)

    by mirko ( 198274 ) on Monday May 10, 2004 @04:14PM (#9110772) Journal
    Another OS which proposed a very ergonomical approach to file system design and implementation was RiscOS [riscos.com].
    Check its Programmer Reference Manuals if you can find these.

  • BeFS lives (Score:5, Interesting)

    by no reason to be here ( 218628 ) on Monday May 10, 2004 @04:21PM (#9110824) Homepage
    OpenBeOS [openbeos.org]'s clone of the Be File System has been selected recently by the folks creating the SkyOS [skyos.org].

    The BFS replacement has been one of the fastest progressing parts of the OpenBeOS project. Dominic Giampaolo has actually commented to the team lead of OpenBFS, and complemented the team on the good work they accomplished.

  • Sky Operating System (Score:3, Informative)

    by Anonymous Coward on Monday May 10, 2004 @04:49PM (#9111092)
    The BFS lives on in a few other places, most notably in a rebuild by the OpenBeOS team. This file system is also being used by SkyOS [skyos.org] for their new file system, SkyFS. They are already making use of the attributes in a number of ways.

  • At Apple Now (Score:2, Interesting)

    by wintahmoot ( 17043 )
    One thing that didn't get mentioned is that the author of this book is at Apple today, and rumor is that he is working on Apple's version of WinFS, a metadata-driven filesystem.

    Be saw the potential of metadata on the filesystem level before everybody else did.

  • Alive? (Score:3, Interesting)

    by moosesocks ( 264553 ) on Monday May 10, 2004 @05:08PM (#9111239) Homepage
    It's sort of hard for an OS to die which wasn't really alive to begin with.

    That being said, I remember using the last BeOS Personal Edition (the one that ran on top of FAT32).

    It was incredibly cool. There's definitely something to be said for an OS which boots almost instantaneously.

    The inability to print was somewhat of a drawback...

  • Explicate? (Score:4, Funny)

    by FireBreathingDog ( 559649 ) on Monday May 10, 2004 @05:37PM (#9111506)
    His style is very straightforward - introduce idea, explicate idea, summarize idea.

    Isn't it ironic (dontcha think?) that 'explicate' appears in a sentence about straightforward writing? How about explain???

  • HERE IS ANOTHER GOOD MIRROR!!!! (Score:3, Interesting)

    by danielblair ( 637546 ) <javahax0r@gmail.com> on Monday May 10, 2004 @05:49PM (#9111614) Homepage
    Here is a link to another GOOD MIRROR! Use it!

    You are all welcome!

    http://www.realcoders.org [realcoders.org]

    Have Fun Everyone!

    -Danny

    joecamel (at) realcoders (dot) org

  • just remember... (Score:5, Interesting)

    by hak1du ( 761835 ) on Monday May 10, 2004 @06:11PM (#9111812) Journal
    Access control lists, user-defined metadata, indexing at the file system level, and all that are not new ideas; they go back to the 1960's. Be didn't invent them, and neither did Microsoft (with WinFS).

    All that complexity comes at a price. UNIX was a reaction against putting so many features into the kernel, and, in my opinion, the UNIX arguments against putting those kinds of features into the kernel are as valid today as they were 30 years ago.

    Unfortunately, the book gives very little historical perspective. It seems to simply assume that "more features" translates into "more advanced". From a quick perusal, division of functionality between kernel and user space seems to be not covered. File versioning at the file system level, another important feature, does not seem to be covered. Historically important file systems and functionality, like those found on VMS, IBM mainframes, and database-based file systems are hardly covered at all.

    This book may give you a good idea of what kind of thinking went into the design of the BeOS file system, but it doesn't even come close to a book on file system design in general. And even as a book on the BeOS file system design, it tells you as much about what the creator of the file system didn't think about as what he did think about.

  • Funny Be IRC quotes (Score:3, Funny)

    by Gunfighter ( 1944 ) on Monday May 10, 2004 @11:24PM (#9113930)
    A few years back, one of the members of my Quake clan was a programmer who preferred BeOS as his platform of choice for development and other everyday tasks. He eventually went to work for Be and we didn't hear from him much after that. Nevertheless, we always gave him hell about his BeOS preference. Here are a few choice quotes from our IRC logs:

    This first one is particularly applicable as it pertains to the "uncorruptable" BeOS filesystem.

    <dEad{Ni}> but you have more problems with win95 than i have ever imagined anyone having
    <Tolen{Ni}> nah...you should see some of the people on my dorm floor...
    <Tolen{Ni}> one guy had to fdisk like 5 times last semester
    <Magaera{Ni}> hehe
    <Magaera{Ni}> You CAN'T corrupt the BeOS file system
    <Magaera{Ni}> Even by kicking out the power cord
    <Gunfighter{Ni}> you can't play Q2 on it either :P

    <Magaera{Ni}> potty stop - brb
    <Gunfighter{Ni}> overkill.. yellow card
    <Magaera{Ni}> what, you'd rather say i was going to "the little programmer's room" or something??
    <Deathwish{Ni}> I got take a BeOS

    <Magaera{Ni}> "BeOS combines the best features of all the major operating systems: the ease-of-use of the Macintosh, the power and flexibility of Linux, and Minesweeper from Windows."

Slashdot Top Deals

Every nonzero finite dimensional inner product space has an orthonormal basis. It makes sense, when you don't think about it.

Close