The Adventures of Systems Boy!

I've been using Time Machine for a while now. And I've noticed some interesting things about its behavior. Of particular note, I've noticed that Time Machine does not back up your data when you are logged out. I found this strange until I figured out why this is the case.

I first noticed Time Machine not backing up logged-out users after setting up the staff computers here at work. Oddly, my work computer did back up when I was logged out, which I realized when I noticed a backup failure due lack of drive space. According to the Console logs this backup attempt had occurred in the middle of the night. Clearly Time Machine was able to backup when users were logged out, but it would only do so on my machine. So what was the difference?

By default, Mac OS X wisely un-mounts external volumes when a user logs out. This makes sense for a number of reasons, not the least of which is the fact that it's what users expect, and it's the least likely to break something if a user logs out and pulls their firewire plug without ejecting their disk. It's a very sane default that errs on the side of data protection. But it's not always what you want. For instance, say you have network shares, like external RAID drives, that are connected via firewire (which, in fact, we do). Or say your network backups that run in the middle of night get stored on a firewire volume (which ours do). If you want these drives constantly available you need to be able to keep them mounted even when no user is logged in. Fortunately, Apple provides a method for doing this, though it's by no means obvious.

The trick to keeping external drives mounted after a logout lies in a little .plist file. The name of this file is autodiskmount.plist, and it does not exist by default; you have to make it. In the file should be the following text:



Put this file in:
/Library/Preferences/SystemConfiguration/

And reboot. (Yes, reboot.)

Now all external drives (firewire, USB, eSATA, etc.) will stay mounted after a logout. If they're shared, they'll always be available. And they'll always be available to Time Machine.

So the difference between the staff machines and mine? My computer is set to never unmount drives at logout. Apparently, Time Machine is perfectly capable of running even when no one is logged in. But it obviously needs the Time Machine drive available to do so. Keeping firewire drives mounted post-logout will allow Time Machine to work all night long. Sweet!

And since I'm so crazy with the Installer Packages these days, I'm including one here that will install the necessary preference file to make all this happen. You know, just to make your lives a little easier.

Download KeepExternalDisksMounted

You're welcome!

UPDATE 1: A reader asked in the comments how I came to have the preference file installed on my system. I'd put it there long ago because I needed firewire drives mounted for rsync backups of staff machines. But I certainly didn't figure out how to create that file myself. Credit for that goes to this Mac OS X Hints hint. It's got all the details if you're interested.

UPDATE 2: One other thing I forgot to mention: Why is this useful? I mean, if you're logged out you're not really capable of creating any new data, so there's nothing really new to backup anyway, right? This is mostly true, indeed. But imagine your boss uses Time Machine for his hourly backups. Now imagine he creates a whole buttload of data — I don't know, emails to the CEO, photos of his kids, whatever — and he creates this data right before he leaves for the day. Then, safe in the knowledge that Time Machine's got his back, he logs out and goes home for the weekend. That weekend there's a power surge or something, and his machine is fried. "No problem," he thinks, "I have my backup." But his most recent data is gone. His photos, his draft to the CEO, gone. And guess who's to blame? Yup. The Systems Admin. Your ass is grass, and Time Machine is the lawn mower. (Uh, this is why I don't write in the mornings.)

Personally, I think it would be smart if Time Machine asked you at logout if you'd like to make a backup, or at least warned you that backups would not be performed after logout. This seems like a bit of an oversight on Apple's part.

The other time this can be useful is when you're creating your first backup. This is typically a lot of data. Here in the office we told folks to let it run overnight. But they couldn't log out. So we dropped them to the Login Window with Fast User Switching. Still, it would have been that much more intuitive if we'd just told them to log out like they always do, and that their backups would be ready in the morning.

So yeah, not earth-shattering, but still potentially useful. And interesting on an academic level to know that Time Machine will run sans login.

I have to go install that preference file on my staff machines now. Bye!

Labels: Applications, Hardware, Lab, Leopard, MacOSX

So far this NetBoot/NetInstall thing is working out a thousand times better than I ever thought it would. I wish I'd done this years ago. Not only does it save time, it also reduces errors. This is often one of the most overlooked features of automating a process: the less human interaction in the process, the fewer mistakes can be made. I have only to compare the set of instructions I gave to last year's crew for building a new system to the instructions for using the new NetInstall system to see evidence of this truism. The list of human actions to take — and, thus, potentially screw up — is significantly shorter using the new process. And that's a beautiful thing.

At this point I've converted about half the staff to Leopard with the NetInstall system, and for the most part it's been quick and painless for both me and them. Contrast with years past, where upgrading staff computers — which are both the most customized, and the most important to preserve the data of — has been fraught with tension and minor hiccups. This year I almost feel like I've forgotten something, it's been so easy. But staff would surely let me know if there were problems. (I'm so knocking wood right now.)

I've also had an opportunity to test building multiple machines simultaneously. Yesterday I built five Macs at the same time, and, amazingly, all five built in about the same time it takes to build one — about a half an hour. I'm astounded. We should be able to build our new lab workstations this summer in a day. And still have time for a long lunch. And for the most part I'll be able to offload that job to my assistants.

As I finish up the system, I've realized some things. First of all, it sort of reminds me of software development — or at least what I imagine software development to be like — because I'm building little tiny components that all add up to a big giant working whole. Also, as I write components, I find myself able to reuse them, or repurpose them for certain, specific scenarios. So, in a sense, the more I build, the easier the building becomes, as I imagine is true in software development. Organization is also key. I find myself with two repositories: one contains the "build versions" — all the resources needed to build the packages — and one contains the finished products — the packages themselves — organized into something resembling the physical organization (packages for staff computers in one area, packages for workstations in another, for instance). It's shockingly fascinating to work on something like this, something that's built from tiny building blocks and that relies very heavily on good organization. I'm really enjoying it so far, and I'm a little sad that the groundwork is built and it's nearly done. There's just something fundamentally satisfying about building a solid infrastructure. I guess that's just something I innately like about my job.

The next step in this process, as I've alluded, will be to do a major build, i.e. our new batch of workstations when they come in the summer, and an update of all our existing computers — all-in-all about 40 machines. Between now and then there are sure to be some updates, so I'll probably update my base config before we do the rest of the lab. And then will come the fun. I will report back with all the juicy details when that happens, in what will probably be the final installment of this series.

See you in summertime!

Labels: Lab, Leopard, MacOSX, NetBoot, Server, Systems

So this is going great. I have a really solid Base OS Install, and a whole buttload of packages now. Packages that set everything from network settings to custom and specialized users. I can build a typical system in about 45 minutes, and I can do most of the building from my office (or any other computer in the lab that has ARD installed).

I'm also getting fairly adept at making packages. A good many of my packages are just scripts that make settings to the system, so I'm getting pretty handy with the bash and quite intimate with dscl. But, perhaps most importantly, I'm learning how to make all sorts of settings in Leopard via the command-line that I never knew how to do.

The toughest one so far has been file sharing. In our lab we share all our Work partitions to the entire internal network over AFP and SMB. In the past we used SharePoints to modify the NetInfo database to do so, but this functionality has all been moved over to Directory Services. To complicate matters, SAMBA no longer relies simply on standard SMB configuration files in standard locations, and the starting and stopping of the SMB daemon is handled completely by launchd. So figuring this all out has been a headache. But I think I've got it!

Setting Up AFP
Our first step in this process is setting up the share point for AFP (AppleFileshareProtocol) sharing. This wasn't terribly difficult to figure out, especially now that I've been using Directory Services to create new users. To create an AFP share in Leopard, you use dscl. Once you grok the syntax of dscl it's fairly easy to use. It basically goes like this:


The "Data Source" is the thing you're actually operating on. I like to think of it as a plist entry in the database — like a hierarchically structured file — which it basically is, or sometimes I envision the old-style NetInfo structures. To get the needed values for my new share, I used dscl to look at a test share I'd created in the Sharing Preferences:


The output looked like this:


Okay. So I needed to use dscl to create a record in the SharePoints data source with all these values. Fortunately, the "sharepoint_group_id" is not required for the share to work, because I'm not yet sure how to generate that number. But create the share with all the other values and you should be okay:


This series of commands will create a share called "My-Share" out of the drive called "HardDrive."

After modifying the Directory Services database, it's always smart to restart it:


And we need to make sure AFP is running by starting the daemon and reloading the associated Launch Daemons:


Not the easiest process, but not too bad. SMB was much tougher to figure out.

Setting Up SMB
Setting up SMB works similarly, but everything is in a completely different and not-necessarily standard place. To wit, Leopard has two different smb.conf files: one that's auto-generated (and which you should not touch) in /var/db, and one in the standard /etc location. Fortunately, it turned out, I didn't have to modify either of these. But still, it led to some confusion. The way SMB is managed in Leopard is rather roundabout and interdependent. Information about SMB share is stored in flat files — one per share — in /var/samba/shares. So, to create our "my-share" share, we need a file named for the share (but all lower-case):


And in that file we need some basic SMB info to describe the share:


Next — and this was the tough part to figure out — we need to modify one, single, very important preference file that basically informs Launch Services that SMB should now be running:

This command modifies the file com.apple.smb.server.plist in our /Library/Preferences/SystemConfiguration folder. That file is watched by launchd such that when it is modified thusly, launchd knows to start and run the smbd daemon in the appropriate fashion. Still, for good measure, I like to reload the LaunchDaemon for the SMB server by hand. Don't need to, but it's a nice idea:


That's pretty much it! There are a few oddities: For one, the new share will not initially appear in the Sharing Preferences pane, nor will the Finder show it as a Shared Folder when you open the window.



But the share will be active, and all will be right with the world after a simple reboot. (Isn't it always!) Also, if you haven't done it already, you may have to set permissions on your share using chmod in order for anyone to see it.

I was kind of surprised at how hard it was to set up file sharing via the command-line. But I'm glad I stuck with it and figured it out. It's good knowledge to have.

Hopefully someone else will find it useful as well.

Labels: Lab, Leopard, MacOSX, NetBoot, NIX, Server

I've become quite the package whiz, if I do say so myself. Actually, I'm probably doing something ass-backwards, but still, I wanted to share some of my working methods as they seem to be, well... Um... Working...

One of the things I'm doing is using packages to run shell scripts that make computer settings (like network settings and user-creation) rather than actually installing files.



This can be done in PackageMaker by taking some creative liberties with preflight and/or postflight scripts. The only hitch is that PackageMaker insists that you install at least some files onto the target system.



So the majority of my packages contain only a single script. That script first gets installed to /tmp, thus fulfilling PackageMaker's "must install files" directive.



The script then runs as a posflight script, and the last line of the script deletes the instance of the script in /tmp, just for good measure.


It could be argued that there's no reason to create packages from scripts, that you could just as easily run the scripts directly in ASR, but packages offer a couple of advantages. For one, packages leave receipts, so it's easy to check and see if something's been set on a computer. For two, packages are easy to deal with; assistants and other SysAdmins know how they work and can easily understand how to use them. Need to change a machine's settings? Don't run a script. Hell, don't even bother opening System Preferences. Just open and run a package. What could be easier (and less error-prone, I might add)? From an ease-of-use perspective, packages have a huge advantage. And ease-of-use adds efficiency. Which is why I not-so-suddenly find myself in the envious position of being able to build systems in about half the time (or less!) it used to take. That's a huge improvement!

Using this method (and sound DNS) I've been able to write packages that configure network settings, create computer-specific users, set custom disk and file permissions, set up autofs, bind to our authentication server and set up SSH for password-less login.

Next on the list: File Sharing!

Should be fun.

Labels: Applications, Lab, Leopard, MacOSX, NetBoot, Server

I recently had occasion to do some Mac OS X DNS configuration. Primary DNS setup is pretty straightforward, but I always get stuck on the secondary DNS config, and this week I had the (not so) rare privilege of discovering I'd been doing it wrong all along. I thought I'd post some quickie instructions for next time this comes up.

So, to set up your Mac OS X Server as a secondary DNS server (that is, if the primary goes down, this one will act as a failover), you'll need to open Server Admin and navigate to the DNS section (we'll use the 10.4 version, but the basic gist is the same in 10.5). Next, add a secondary zone under "Secondary Zones." This first one will be for your forward lookups. In the first field add the full name of your zone. In the second field add the IP address of your primary DNS server. It should look like this:



Now add another secondary zone for the reverse lookups. This is exactly the same except that instead of the zone name in the first field, you'll use the reverse zone information. It looks like this:



Note that the reverse IP in the field should cover the zone IP range. That is, if your primary DNS server covers "192.168.1" zone, you'll enter the reverse info for that range. If your primary DNS covers all of "192.168" (notice the truncated IP number) then you should enter that. All together it should look like this:


Once this is all set up and saved, you can start the service. You can tell it worked by listing the contents of /var/named. You should see two new files, one called your.domain.com.bak and one called your.IP.in-addr.arpa.bak. And these should be populated with the same DNS info as your primary.

And that's it! Enter the new secondary DNS server info in your Network System Preferences, just after the primary, and if your main DNS fails you'll be covered.

Oh, almost forgot, thanks to this dude for helping me figure all this out.

Labels: Lab, MacOSX, Server

So here's the plan, as it stands right now. (Yes, I have a plan already! Yes, that was quick.) First, build an image that's good for all workstations (laptops, staff machines, standard workstations, etc.) throughout the lab with:

• Base OS (Mac OS X Leopard 10.5)
• A DHCP network connection
• Apple applications
• Adobe applications
• Drag-and-drop applications
• Other third-party applications
• One admin user
• ARD active

IMAGE THIS SYSTEM
This becomes the base system build, the Master Image — the replacement system if a machine ever needs rebuilding. It is the only full system image. There is only one.

Everything else that is machine-type specific — that is, users, applications, application components, crontabs, anything — gets turned into and installed via either a downloaded or hand-rolled package. So far I've been using Apple's PackageMaker, which has matured a great deal since last I tried it. It's pretty nice. I'm also taking a look at Iceberg, which also looks to be pretty full-featured and nifty.


These packages can be machine-type specific and stored in a simple folder hierarchy by machine type — laptop packages in the laptops folder, etc. — for organizational purposes. In addition to being machine-type specific, packages can also, I believe, be machine specific. That is, I think we can make settings like computer name and network settings using packages as well. So what we're talking about here is a system of computer building that happens completely over the network, and that can be directed almost entirely from one ARD-toting computer, that computer being mine, of course. (I've always said, the sign of a good SysAdmin is that he never leaves his chair.)

I've gotten pretty good at making application packages, at this point (not that it's terribly hard, mind you). My next step will be to learn how to make system settings with packages as well. My other next step is going to be, of course, creating the Master Build. None of this building happens 'til summer. But still, something tells me it's going to be smart to start this process now and see what crops up over the next couple of months.

As usual, I'll be reporting any new and/or interesting developments.

Oh, and thanks to everyone who commented on the last post. The comments were extremely useful!

Labels: Lab, Leopard, MacOSX, NetBoot, Server

My big, fat, self-assigned new project — or, as I like to call it, the bug up my butt — is system roll-outs. That is, I realized at some point two things. One, I am managing way more computers than ever before, and way more than I realized; and two, the scope and variety of these various systems has become increasingly wide. These realizations inevitably brought me crashing, headlong (yes, headlong) into a third and final revelation: I need to come up with a better system for managing machine builds and systems roll-outs.

Enter: NetBoot.

Actually, let me first explain how we've handled this in the past. When I began this job we had maybe 15-20 Macs running OS 9 briefly, and then OS X since about its inception. Mac OS 9 was notoriously easy to build. Just copy that shit and be done with it. But Mac OS X was a different beast entirely. Mac OS X was complicated. Moody. A tougher nut to crack. Mac OS X required me to delve into the dark arts of system cloning.

Cue thunder clap, scary music.

The process that eventually evolved was cloning over firewire. We'd build our master machine — a basic Mac OS install with all the latest updates and our requisite software — and then clone that to the other machines over firewire, with clients booted into our Master via firewire target disk mode. This was a quick and dirty way to build a bunch of systems. Once the group was built we could customize machines or groups of machines as we saw fit. For a lab of 15-20 Macs this has worked swimmingly. But our lab has grown slowly and steadily, and completely without my realizing it.

My latest count shows our lab at more like 50 Macs now. We've added a bunch of stuff: laptops, servers, more A/Vmachines of various configurations, a render farm, and of course more workstations. Using our old system of firewire cloning is becoming increasingly clumsy, slow and error-prone. We need a better way of doing things.

Okay, now. Enter: NetBoot!


NetBoot is Mac OS X Server technology that allows for centralized storage of and access to system images for installation over the network. The way it works is this: You build a system, trick it out, make it perfect — this is your Master System. Put it on a firewire drive or something transportable, because you can't be booted from your Master System for the next series of steps — imaging. Run the application called System Image Utility, that comes with Mac OS X Server's server tools, and create a NetInstall image from your Master System. Load that image onto your server and enable it in the NetBoot settings in Server Admin. And what happens next is something akin to magic (unless you use Linux, and then it's pretty par for the course, I guess.)


With your NetInstall image enabled on your server, go to any client system and open the Startup Disk Preferences. Where you'd normally see the "Network Startup" icon (I'll bet you always wondered what that was for), you'll now get something a bit more descriptive. You should now be given the opportunity to boot from your master image. Choosing to do so will incur further magical results. Your system will now boot... Over the network! What's even cooler is that you'll be booted into the same basic installer environment you'd see if you were booted off a Mac OS X install disc, and you'll be walked through the steps required to install your Master System onto your computer. Um... OVER THE NETWORK!

There are some immediate advantages to a system such as this. First off, you can have a bunch of different Master System images for various build configurations in your lab. For instance, we can have a separate build for laptops and desktop machines. Cool! Also, this can all be automated (thanks to Automator integration) to make the process run almost entirely unattended. Sweet! And since the whole thing sits on the server and is available at all times, if a machines needs a rebuild, you just set it and forget it. Awesome!

But NetBoot has its drawbacks as well. Images — particularly large images — take for-freaking-ever to build. To give you an idea of how long, my near-36GB boot drive took an hour or so to clone to firewire, then 3-4 hours to be imaged by System Image Utility. So each build will take several hours to create. And God help you if you make a mistake on one of your images: the NetInstall images are read-only and can't (to my knowledge) be modified once they're built. NetInstall technology can't be used for non-package installers or updates either (again, to my knowledge), so you'll have to run all your Adobe and Microsoft updates by hand as usual. Also, one minor caveat that threw me at the outset: Mac OS X 10.5 server can only serve Mac OS X 10.5 builds. So you'll either be needing to update everything to Leopard or wait until your server and client OSes match.

I think I'm right on the crux of really needing this. I could probably live without it. Keep doing what I'm doing. But I do think I'm at a point where the benefits of using NetBoot outweigh its limitations. And I have enough resources now (like the required massive drive space and robust stable servers) that it's not impractical on the physical level. So, this summer I plan to use NetBoot to build my lab.

I mean, imagine this: you get your new systems over the summer, you unbox them, plug them into your network, set them to NetBoot (everybody say "command-n"! Good!) and go home for the night. You come in the next day and everything's basically done. You've just built your lab. Overnight. In your sleep. I don't know. Sounds pretty cool to me.

I'll let you know how it goes.

Oh, and by the way, in case you can't tell, I'm just now learning all this. It's still very new to me and there is a lot about it I don't know. So if anyone has any experiences or insights into using NetBoot (or if I get any of the facts wrong), I would absolutely love to hear about it. Please feel free to share your experiences in the comments.

Labels: Lab, Leopard, MacOSX, NetBoot, Server

So there's an incredibly annoying and puzzling behavior in Mac OS X with regards to duplicate computer names on the LAN. Most Mac Admins probably know what I'm talking about. Here's the deal: Let's say you have a Mac Pro on your home network named Spanky and it has an IP address of 192.168.1.25. Let's also say, for the purposes of argument, that your best friend — who likes to emulate you in every way — has a Macbook named Spanky with an IP of 192.168.1.25 on his home network. (Hey, it could happen.) Now let's say your pal decides to come over, and he decides, "Hey, I think I'll bring my Macbook over so we can swap some illegally obtained music and pornography." He gets to your house to find you happily surfing the 'net. He whips out his Spanky and plugs it into your network, fires that puppy up, and Bam! All of a sudden your Mac Pro locks up. You can no longer surf. And you get an error message that looks a little something like this:


You go to your Sharing Preferences (or your Terminal, or what-have-you) and, sure enough, your computer has been renamed. Renamed! WTF! Why is your computer, on your network, suddenly called "Spanky-2"? Because that's how Mac OS X handles duplicate computer names on the same network. It renames the existing computer to existing computer-2. Not the intruder. Not the new kid on the block. Your computer is now the computer formerly known as your computer. Why, it's pure genius, I tells ya! Brilliant!

Seriously, what in God's name were they thinking? Because, the way I see it, this goes beyond annoying into the realm of the dangerous.

Case in point: Let's say your Mac Pro Spanky is actually a server that provides services — authentication, Kerberos, LDAP, file sharing, the works — for a network full of computers. And let's say your friend is actually a guest on that network. When that guest plugs his computer into your network, and it just happens to be named the same thing as your server, God help you. You just lost — well, I'm not sure how much, but — a significant portion of your services. And all it takes is a computer name? I can rename any Mac on any network from any other Mac on that network by just changing my Mac's name? What's more, if you can get the IP of that server, you can bring it down entirely. That is total shit.

Strangely, it's been this way for at least three iterations of the OS now — since 10.3 — and it's still this way in 10.5. I am appalled. Can someone please explain the rationale for this to me? Please? 'Cause from where I sit, this is a major security flaw. To my mind it makes way more sense to have the newly installed machine make changes to it's configuration than to essentially be able to force changes on another machine. It's just backwards. And dangerous. And it desperately needs to be fixed.

UPDATE 1:
Mat X points out two very important facts: 1) the name change in this instance should only affect the Bonjour (.local) name of the duplicate machine, not it's actual name (the name it calls itself) or it's FQDN (the name as resolved by a DNS server), and 2) a client name change on the LAN should not be able to bring down a server with the same name because of the previous fact.

Dude, you're totally right, though I have seen, way back in the old days (10.2 maybe?) a duplicate name kill a server. So it did used to be possible. Nowadays though, Mac OS X Server has better, smarter naming conventions that prevent such things. I will say, though, that what prompted this was that I booted a clone of my own machine up on the LAN (same name, same IP) and it killed my computer, internet-wise, probably more because of the duplicate IP address. It's possible that all would have righted itself over time if I'd waited to see. I was just so annoyed by the behavior that I went on a bit of a rant. I still think it would be better behavior to leave the existing LAN client alone and make changes only to the new Mac on the LAN. But still, I got a little carried away.

Thanks for keeping me on my extremely bitchy and unscientific toes!

UPDATE 2:
So I've been doing some more testing on this issue. And while duplicate names on the network will not bring down your Mac OS X Server, a duplicate IP address will. Here's what I did:

1. I changed my client's IP address to match that of the Mac OS X Server (both Leopard 10.5.1). My machine got kicked off the internet and I got this warning:
   
   
   
2. I rebooted my client machine. Once the client had rebooted, the server got the above alert.
3. At which point the server could not ping out to the internet. Bad Mac OS X! Bad!

So, while my rant is partially in error, there does seem to be a bit of a flaw in the way Mac OS X handles new duplicate clients — particularly duplicate IP addresses. And I maintain that a better way would be to only modify the behavior on the most recent addition to the LAN.

Labels: Lab, Leopard, MacOSX, Rants, Server, Systems, Tiger

I recently wondered aloud about the benefits of MacFUSE-supplied ssh access in a web development environment. A reader asked for a follow-up in the comments to that post, so I thought I'd just do a quickie today.

After writing the article, I tried out MacFUSE and the sshfs extension to that bundle. This combo facilitates the mounting of ssh-enabled computers directly on the Mac OS Desktop via a bundled GUI app, foregoing the need for an FTP client application. It was our hope that this would provide a more intuitive and user-friendly experience for our web developers. Unfortunately, in my experience there were bugs: the ssh filesystems often hung or had trouble mounting, often requiring a logout or restart to get things back to normal. (Mind you, this was over a month ago, in Mac OS X 10.4.something-or-other and God-knows-what version of MacFUSE and sshfs.) Also, from a user-simplification standpoint I felt that adding yet another application to the mix (the sshfs GUI app for mounting shares) didn't really do all that much to simplify things. In fact, in some ways, it complicated matters as it is not standard practice anywhere as yet.

That said, I really like the idea of mounting any sort of network share in the Finder. It's what's done in every aspect of our filesharing lives except web development, for some reason (that reason mainly being that Apple, inexplicably, still has not implemented this capability into the Finder.) As I mentioned in the original article, there are styles of development that take place outside the lab that are inherently different from what can happen inside a lab. The use of MAMP, for instance, is not friendly to a shared, multi-user environment, but is perfectly suited to a lone user on a single computer. So, again, I wonder aloud, do we provide a different way of working inside the lab than outside? A completely different workflow?

Ultimately, my answer will mostly be yes, I think. We provide it. We provide it right alongside the sorts of workflows that are common in the industry today. We provide both. We show students. We show teachers. And hopefully, someday, down the line, this will become the standard way of developing for the web. Until then we leave the old ways in place; they are not mutually exclusive.

My concern with MacFUSE and sshfs right now is stability. Because the product requires kernel extensions, and because I saw bugs in my limited testing, I am reluctant to put this on our machines. I can, however, use NFS from within our lab to provide a testing ground for the same sort of behavior that MacFUSE provides (i.e. ssh servers on the Desktop). And this will actually fit in nicely with the way we mount other shares in the lab as well (more nicely than the sshfs GUI, in particular). I will most likely implement this over the summer, however, as there's not enough time in the semester for it to really have much impact. Also, things will probably change a bit with the arrival of Leopard to the lab, and I don't see much point in making folks learn it twice.

Again, external development can be handled differently. So I may recommend, for advanced users, the use of MacFUSE outside the lab if they really end up digging the Desktop web development experience.

That's my take at this point on MacFUSE and web development. Hope it's useful to someone.

Labels: Applications, Internet, Lab, Server

Our home account server — the one all our network users' home accounts live on — mounts via NFS. I've already mentioned one of the improvements with regards to this behavior in Leopard, namely, autofs. But I just discovered another.

See, that NFS mount I'm always talking about, well, it's a Linux RAID — Fedora Core 6, I believe. And we've put quotas on every user's account to keep it from filling up. This has worked great, for the most part. The problem has always been how the Tiger Finder handled a user exceeding his quota. Basically, what would happen is that any Finder copy that exceeded a user's quota would fail, mid-copy, with an extremely vague and essentially — unless you happened to know you were looking for a quota problem — useless error message. Can't recall it exactly, but it's something along the lines of: "The Finder encountered an error and could not complete the request."

Ew.

The good news is that the Leopard Finder behaves exactly like we'd want and expect:


This alert actually came somewhat prematurely, as I hadn't actually exceeded my quota, and the alert seems to be triggered by hitting the soft quota rather than the hard. What's up with that?

Still, I have to say, I'm endlessly impressed with these little tiny details. They make all the difference in the world from an admin standpoint. I no longer have to worry about quota weirdness! It's great to see Apple do so much for admins with this release. I'm seriously pleased as punch.

Labels: Lab, Leopard, MacOSX

I'm not a web programmer by any means. But a component of the department I work in deals with the web from an artistic standpoint, and a subset of that group does, in fact, do programming. We have web programming classes.

Recently, one of the teachers of one of those classes made the charge that our approach to web programming is old and outmoded. Whether this is true or not is not really the issue. I've been looking for new ways to think about the systems end of that workflow because, well, that's my job, and because it's an inherently interesting challenge to me. How can we make our web development environment more user-friendly?

One general suggestion has been to make the experience more "OS-like." And one step in this direction is to have the web server mount on the Desktop, allowing the developer to work on her site as if it were local. That is, rather than firing up one of the popular SFTP clients and transferring files back and forth from the local machine to the server, the developer could mount her site — or actually, the share her site lives on — directly on the local filesystem. I have two options here: MacFUSE and NFS. I'm testing both currently. So far I've had a couple minor hiccups with MacFUSE's sshfs.app, but it looks to be a fairly smart and user-friendly implementation that web developers here might benefit from. And the NFS approach would work well also, though only from inside the network.

I'm curious what other Lab Admins are doing with regards to web development in their environments. How have you facilitated ease-of-use for a process that's inherently complicated? Or have you? Also, I'm curious if anyone is using MacFUSE — specifically the sshfs component — and what experiences you might have had with it, either positive or negative.

If any of you fine readers have any thoughts on this I would really love to hear them. I've been querying students, staff and faculty for ideas, but haven't come up with much. Maybe things here are perfect, but somehow I doubt it. And, as always, I really just want to make things better.

Please sound off in the comments if you're so inclined.

Labels: Applications, Internet, Lab, Server

The founding article for this blog — indeed, the very reason for TASB's existence — was a little post called "Tiger Beefs" in which I ranted for a few thousand words about everything I disliked about Tiger. It's been over two years since that faithful first post, and Apple has just released the follow-up to Tiger, Mac OS X 10.5, codenamed Leopard. Please note the absence of the word "beef" from today's title. So far, I have to say, I'm quite pleased — and certainly not deeply irritated — by this latest OS iteration.

Right off the bat I want to point you to the best and most thorough review of Leopard. Every time a new cat is born, John Siracusa not only reviews many of the new features, but goes deep into the depths of the OS to tell us geeks what's really changed and what it means for the future of the platform. It should be required reading for anyone seriously interested in Mac OS X changes.

Also, I want to point you to Apple's infamous list of new features. It's pretty comprehensive for the surface features, and even touches on some of the things I'll deal with here. And speaking of, my particular perspective on Leopard will be less about productivity features (though there will be some of that, to be sure) and more about Leopard from an administration and maintenance standpoint. So, let's get started!

Time Machine
The most highly touted new feature in Leopard — and rightly so — is Time Machine, which automatically makes backups of your data to any external hard drive (or even, I'm told, partition). The whole idea behind Time Machine is that it's so simple, and requires so little thought, that anyone can — and, more importantly, will — use it. It's backups for the masses. And while Time Machine is really made for the end-user, the fact that such a beast now exists as part of the OS is a huge boon to SysAdmins.


I maintain a backup system for all staff members in my department. Anyone who's ever had to deal with such a system knows what a pain it is to implement and maintain. In the old days, we used to back up to tape using Retrospect. But as data storage became increasingly large, and tape increasingly expensive, the system grew unwieldy. An unwieldy system, as you surely know, is not reliable. A few years ago (in fact, with the introduction of Mac OS X, come to think of it) we moved to the free, scriptable, and very reliable rsync (we use the RsyncX version). This allows us to back up over the network to a large RAID drive. But still, the scripts require occasional maintenance, staff must be sure to leave their computers on. There are numerous points of failure. And most inconvenient of all, if a staff member does lose data, they have to come to me to retrieve it, which is inconvenient for both them and me.

Time Machine removes that last step from the equation. Time Machine puts the end-user in control, not just of their current data, but of their backups as well. Now, if a staff member accidentally throws away a file, or makes changes they don't like to a document, or whatever, they simply activate Time Machine and roll back. No freak-outs. No calls to the SysAdmin. No worries. Time Machine is frickin' beautiful.

I will continue to make backups to the RAID with rsync for the foreseeable future. It doesn't hurt to have an extra backup, and, Hell, the system's already in place. But I've also bought all staff a firewire drive specifically for Time Machine as well.


One thing to note about Time Machine: It is geared towards the idea of backing up everything. Like in Spotlight, you can add exclusions to Time Machine, but the default is to back up all your data. A fellow SysAdmin complained that he needed the ability to select what would be backed up, not what wouldn't, if this were to be useful in a production environment. Yes, my friend, but this is not made for production. It's made for people. So the default is, back up everything. What could be simpler?

The Finder
I won't spend too much time on the Finder. In a nutshell, I'm mostly happy, though I'm a bit peeved that the first thing I felt the need to do was hack that ugly-ass Dock.


Seriously. Ouch. I can see liking it on first glance. I mean it is shiny. I know people like shiny. But damn is it intrusive, and not the least bit of an increase in functionality. Yikes! What were they thinking?


There are a few awesome new touches in the Finder, though. Quick Look is probably my favorite. Hitting the spacebar to view a preview of a document is a great productivity boon. Students in the art department where I work will love it for presentations as well. It's beautiful, useful and extremely well-implemented. I only wish it were more key-command-able. (Or maybe we'll discover that it is.)


Speaking of key-commands, the Desktop now has a presence and key-command in the "Go" menu (it's command-shift-d). Sweet!


Also, a long-standing (read: never solved) problem with Tiger's inspector, wherein the inspector would not properly update file ownerships, has been fixed.

I also rather like the look of the new Finder. I'm pleased as punch that there's finally a window standard, and that it's not brushed metal. While I'd probably prefer a lighter shade of gray, and apps in the background to be darker rather than switching to a lighter shade (dark recedes; light comes forward, at least that's what they always told us in art school), the current iteration is really quite nice. The Sidebar is also, in my opinion, more efficient than it once was. And Cover Flow in the Finder might even prove useful at some point.

Other nice touches:

• Clicking on a file name only highlights the file's name, not its extension, thus making file renaming a lot quicker and easier.
  
  
• Drop shadows are larger and darker and generally more dramatic, making windows easier to discern.
• Drop shadows are also now included in screen captures of individual windows.
• File sharing, which is now possible on a per-folder basis (hooray!), can be activated and configured right from the Inspector.

One oddity: the Finder seems to be a bit more fascistic about what you can and can't do with your data. In fact, it disallows trashing key folders in your home account. I was unable to trash, or even rename my Library folder from the Finder. This might be great for the home user. But it could slightly complicate troubleshooting from an admin standpoint. Not a big deal, but I'm not crazy about the trend towards over-management of user data. It's fine for Time Machine. Not so sure about the Finder.


Still, that's a lot of good and very little bad. Overall, the Finder's a big win for me.

Disk Utility
Probably the best thing about Leopard is that there is so much good stuff for SysAdmins. Each OS upgrade has brought us a couple goodies, but Leopard is chock full of them, and the goodies are so... Uh... Good...

First off, Leopard now handles broken disks more gracefully. Attach a damaged external firewire drive, for instance, and if it's mountable, Leopard will mount it and allow you to copy any data that might be salvageable. This actually happened to me in the beta days, and Leopard provided successful, albeit partial, disk recovery where Tiger simply refused to even mount the damaged drive. That's a pretty sweet improvement that no one but SysAdmins are likely to see. Kudos to Apple's Disk Utility team for that one!


Another huge advancement in Disk Utility is the ability to re-partition a drive without wiping it, within limits, of course. Actually, it might be more accurate to say that Disk Utility allows partitioning — or splitting — of partitions. Let's say you have two partitions. But you want to turn that into three. In Tiger and before you had to erase the entire drive and repartition. In Leopard, you can cut one of your two partitions in half (or quarters, or whatever). Leopard will even indicate the free portion of the disk and cut it at the right point. It's pretty damn cool, and something I've been wanting for a long time. For forever, really. I've already used it in the beta, and it seems to work great. Cool!


The one caveat to this dynamic partitioning is that the disk must be formatted using the GUID partition map, which Apple has adopted for the move to Intel. It's GUID that makes all this possible. The old style Apple partition map won't allow non-destructive partitioning.


The final touch in Disk Utility — and actually, this appears to be true through much of the new OS — is that the wording of dialog boxes and information panels has been made much clearer. This should do a lot to make scary disk operations a bit less scary.


Directory Utility
The application formerly known as Directory Access gets some love in Leopard too. Now called Directory Utility, the application does more with less. It's simple, four-tab interface still allows the configuration of services, but there's just a lot less to configure. The only services left now are Active Directory, BSD Flat File and NIS, LDAPv3, and Local. Gone are the services that were never really configurable in the first place, save for turning them on and off.


But Directory Utility allows for the configuration of Directory Servers now in a separate panel, and this is where you'll most likely set up your Open Directory server (though the option still exists in the list of services as it always did). Setup is super simple: check the type, and enter the name. That's it!

Directory Utility now also has a panel for configuring NFS mounts. This is also really easy to use. Simply type the path to your NFS server, and type in the mount point. Directory Utility will verify that the server is functioning and then, when you hit apply, it will mount it. Neat-O!

NetInfo (RIP)
NFS mounts were once handled in an obscure admin utility called NetInfo Manager. NetInfo Manager is now dead. Leopard has moved all of its arcane functionality into other more GUI-friendly apps. Directory Utility handles NFS mounts. The Finder and Sharing Prefs handle per-folder file sharing (which was once the domain of a little app called SharePoints, which configured properties in NetInfo). And home account location can now be configured by using the Accounts Preference Pane and control-clicking the account in question, then choosing "Advanced" and selecting the appropriate options. It's true, I can't think of too many more reasons to go to NetInfo Manager.

But wait... How do I activate root?

AutoFS
autofs is the new automounter daemon in Leopard, and boy is it cool. I've watched with envy for years as my Linux counterparts dynamically mounted NFS shares — or folders within NFS shares — as they get called by the OS. I realize that autofs does a great deal of good for hangs caused by network dependencies, but what I'm most excited about is the dynamic nature of autofs. Prior to Leopard we used automount, which I simply could never coax into doing what autofs does out of the box. With automount, we basically just hard-mounted our NFS server at /home at every boot. With autofs, however, we can specify a wildcard in our map file. What that allows us to do is to never keep the entire NFS server mounted, ever, ever. Instead, when the needed share is requested, autofs mounts the portion of it that was requested.

Perhaps an example is in order. Currently, our NFS server gets mounted in its entirety at /home on every client in the lab. This happens using an arcane Startup Item that contains a truly Byzantine script that I made. It's horrible. Not only does it require this crazy-ass script, it only happens at boot or when automount is specifically restarted. It also requires (for reasons I can't recall) a series of symlinks to land in the /home folder properly. And, worst of all, it keeps the entire home account server mounted over the network on every client all the time. Yuck!

By contrast, autofs requires no such Startup Item. You simply edit one tiny text file (/etc/auto_home, if you're interested) and you're done. Not only are you done, though, but the entire process is now dynamic. No reboot required. In fact nothing happens. The home account server doesn't mount... Until it's called! That's right. No home account server is mounted until joe_user comes and logs in. When that happens, autofs springs to life and mounts the user's home account. And here's the other thing: it only mounts the user's home account, not every folder on the share. This is a huge savings in terms of network overhead. It's also much easier for me to maintain and manage. For me, a working autofs is a huge, huge deal, and it's the thing I'm most pleased about. SysAdmins doing any kind of NFS home account mounting will totally understand where I'm coming from here, I'm sure. This is fantastic. My job just got easier, and my network and Mac systems just got a helluva lot more efficient. Awesome!

Other Notables
There's a whole other list of new features that should make SysAdmins and even regular folk pretty happy. Here are my faves, in no particular order:

• Preferences and applications (i.e. Sharing, etc.) that can be applied to specific users now list network users and groups.
  
  
• Login, remote login (SSH), and file sharing are all now configurable on a per-user/group basis.
  
  
• The firewall is now configurable on a per-application basis.
• There is now a built-in guest account that gets deleted at logout.
• Software Update now logs you out for certain updates where your presence might cause problems.
  
  
  
• Software Update now remembers what it's downloaded and will use that if you postpone an update, rather than having to re-download it.
• iCal event entry doesn't suck as bad now, and is reminiscent of Google's method of contextual calendar entry.
• Dictionary now includes Wikipedia and can easily toggle the three views (dictionary, thesaurus and encyclopedia) or view them all at the same time.
• Spotlight works well now, like it always should have.
• Spaces might actually be useful as well!
• Screen Sharing! For free! Cool!
  

So, that pretty much covers my initial impressions of Leopard. We'll be holding off on installing it in the lab until I can run the majority of major applications (currently, AfterEffects is listed as not working, and that's a deal breaker). Until then, I will run it on my test machine.

And happily. Leopard has been extremely stable and reliable so far, and I must admit I really rather like it. I was never a big fan of Tiger, actually. I found everything "cool" about it to be buggy or annoying. Spotlight sucked, Dashboard was stupid, and there were all manner of problems, and few features to recommend it over Panther, at least not from a SysAdmin standpoint. Leopard, on the other hand, is completely the opposite. There are tons of new, useful features for both users and admins alike. So far, I'm very happy with this release.

Nice job, Apple people!

Labels: Applications, Lab, Leopard, Systems

I'm still swamped, buried under the avalanche of work that is the direct result of endless delays to the renovation of our department. Nevertheless, I find myself with little to do today. Or, I should say, little I'm actually able to do. Today's delay, from the "I can't catch a break to save my life" department: Keyboards. In Spanish!

Today I finally got a chance to start unboxing and building my new Macs. Opened up the first one and noticed that the keyboard said "Ayuda" where it should have said "Help." Uh... Shit...

So now what? Will I have to send the entire box back for exchange, or just the keyboards? Until I know the answer to this, I can't build my systems. And I won't know 'til Monday, because purchasing is closed this week.

So I blog.

Hola.

Labels: Hardware, Humor, Lab, Systems

Typically, when you set up a Mac client to bind to a Mac server using Directory Access, there is one lone entry in the Security settings that is checked by default, and that is the "Disable clear text passwords" setting. This seems like a prudent default, and I always leave it checked. I assume that this means that passwords are then sent to the authentication server in some sort of hashed or encrypted form, and that both server and client are set up to negotiate this transaction properly out of the box. Indeed, most of the time this does not present any sort of problem whatsoever.

But for some reason, every now and then, completely randomly, Mac clients will suddenly and mysteriously be unable to authenticate to my Master Authentication Server. Seriously, nothing's changed. Just all of a sudden, Macs can't authenticate. The solution? Un-tick that "Disable clear text passwords" box under the LDAPv3 server configuration's Security tab. Next thing you know, everything's right as rain.


Seriously, can anyone tell me what I'm doing wrong? 'Cause frankly, it's annoying.

Labels: Lab, MacOSX, Server

It's been quite some time since I've been able to post anything of any substance. This has a lot to do with the fact that I've been super busy relocating our department and participating in the gut renovation of our lab. This has been an immensely stressful process, but in the end I find that I've learned so much from it, I simply can't complain. I'm coming out a far better SysAdmin than I was going in. And that's a remarkably valuable thing to both me and my employers.

But moving and planning the physical aspects of the new lab has only been a portion of what I've been working on. This renovation has been the perfect opportunity to rebuild our network infrastructure, and part of said rebuilding has resulted in the near completion of our authentication unification project. At this point we've gone from eight different authentication servers — that is, anytime we created a new user, we had to do so on eight different systems — all the way on down to two. Which means that now, anytime we create a new user, we do so on two machines.

Our goal is to get it down to one, hopefully before the Fall semester begins. Our mail server is proving to be the most difficult machine to get working with LDAP authentication, mainly because it authenticates mail users through the wonders of some weird combination of authd, Courier and PAM, and we've yet to crack the magical code that gets these all working in tandem via LDAP. Aside from Mail, though, everything is done. So I thought I'd take a bit of my hard-earned vacation and loosely describe to you how it's all working.

Before I start I'd like to just acknowledge all the help I've had from my fellow SysAdmins in the department. I had a huge amount of assistance on the *NIX server side of things, as well as with network infrastructure and even some last-minute PHP finagling without which this project would have taken significantly longer. In fact, all I really had to do was build the authentication servers and clearly articulate what I wanted. I'm extremely grateful to everyone who helped out.

The little bit of network infrastructure I mentioned is our DMZ. We now have a proper — and more importantly, properly secured — DMZ on which to place an authentication server. I won't go into too much detail here, but suffice to say, having a secure DMZ gives us all kinds of options for authentication between internal and external networks, and makes me feel a whole lot better about using Mac OS X Server as our authentication system for both networks.

Yes, we are using Mac OS X Server to authenticate our entire network. The reason is because Mac OS X Server is the most mature and usable implementation of LDAP for user authentication available on the market today. Is it perfect? No. Is it completely secure? Probably not. Is there anything that even comes remotely close to being able to handle the complexities of user management and database redundancy across platforms with such remarkable ease-of-use? Nope. Nothing. We tried building our own custom LDAP server, which would have been excruciating, and would have taken forever. We tried Red Hat's Directory Server, which looks like it will eventually turn into something to match Mac OS X Server, but which just wasn't yet up to snuff. Nothing matched Mac OS X Server, which did everything we wanted it to, right out of the box and with a minimum of fuss. In fact, once the user database is built, building a Mac OS X master or replica authentication server is a complete and total breeze. At the time of our building and testing it was really the only practical option.

So, here in a nutshell, is what we have:

Internal Network
All authentication originates from the internal network. Passwords can only be changed from the internal network at this time, which is by design. Systems on the internal network include:

• Master Authentication Server
  Hosts authentication for... Well... Everything, really. This is essentially the same server we used all last year for all our internal authentication needs for Mac, Linux and Windows workstations. It's now being used to push authentication to the external network as well.
• Internal Replica Authentication Server
  This provides replication of the Master. Should the master fail, the Replica is intended to pick up services (though this doesn't always work perfectly).
• File Servers
  We have two file servers on the internal network — a Mac and a Linux box — both of which authenticate directly against the Master.
• Workstations
  We have about 30 Mac, Windows and Linux machines all authenticating to the Master.

DMZ
The DMZ sits between the Big Bad Internet (BBI) and the internal network. It has its own firewall that is fairly strict about what can get in from the BBI. All DMZ authentication originates from the internal network, but is provided by a single server which sits on the DMZ. Systems on the DMZ include:

• External Authentication Server
  This server is also a replica of our Master, but it's not intended as a failsafe. Rather, it provides authentication services to the entire DMZ. It gets its user database, of course, from the Master. But for other systems to bind to an LDAP server, its role must either be "Master" or "Replica." Setting the role to "Connected to a Directory Server" won't work. In addition to sitting on our DMZ, which is properly firewalled against the harsh realities of the Big Bad Internet (BBI), this system also makes use of its own strict local firewall for an extra added layer of security. Also, all replication communication between Replica (DMZ) and Master (Internal) is encrypted.
• Data Server
  In addition to unifying authentication, we've also consolidated data storage and access wherever possible. In the past, for instance, movies streamed from the Quicktime Server were stored on that machine's local drive. Web sites were stored on our web server. So, building a web site that used Quicktime Streaming required users to log into two separate machines — the Web Server and the Quicktime Streaming Server. Now we're storing all user-generated content on a separate, dedicated machine — our Data Server — and sharing that machine out to the various servers via NFS. Centralizing this data store means users have only to log on to one server for anything they ever want to do. And also that only that server needs to authenticate users. And yes, that server authenticates them via LDAP on our External Authentication Server. All neat and tidy. Internal and external home account data is still segregated, however — users still have separate internal and external data storage. Though, if we could figure out how to do it securely, this could change.
• Quicktime Streaming Server
  This machine also uses its own local firewall. It gets its user database from our External Authentication Server over secure channels using the "Connected to a Directory System" as its role currently. Ultimately, however, because of the Data Server, this machine will not need to authenticate users. We are leaving the ability open temporarily to accommodate legacy users.
• Drupal CMS
  Our new Community site is built on the Drupal engine. We're using the LDAP module to authenticate to the External Authentication Server. Drupal's LDAP module is simple and easy to set up, as is the Drupal system as a whole. So far we're very happy with it.
• Computer Reservations System
  This is a custom web app built long ago by a former student. We've (and by "we" I mean my colleague) basically hacked the PHP code to authenticate via LDAP rather than MySQL.
• Mail Server
  Currently not authenticating to the External Authentication Server. We're working on this and hope to have it working by the beginning of the school year.
  

The Future
Yes, there's more we want to do. It's always amazing how, once you've completed something, you immediately start seeing ways to make it better.

• More Redundancy
  Ultimately, in addition to the Replica, I would also like to automate a clone of the Master's boot drive to an external firewire drive as sort of an ultimate safety. Should anything ever go wrong with the Master, I simply plug the firewire clone into virtually any Mac system on the internal network and I'm back on my feet. It might also be wise to have some sort of failsafe for external authentication as well.
  
• More Security
  while our setup is fairly secure right now, there are a few areas I'd like to beef up even more when I get a chance. In particular, our CMS connection is not as secure as I'd like it to be. And ultimately I'd like to harden every machine on the DMZ to the best of my ability.
• More Unification
  Anything else we can unify — and at this point that's mostly internal and external data — I'm open to considering. It's going to be really interesting for me to look critically at what we've done so far and find the flaws and refine the system. But I'll constantly be looking at ways to simplify our current setup even further without compromising security. The easier our network is to use, the more useful it becomes. We've come a long way, but I'm sure we can find even better ways to do things.
• More Services
  Now that we have an infrastructure in place for user creation, we can add services freely to our network without the worry of creating users for said services. New services need only the ability to authenticate via LDAP. We're already planning an equipment checkout system, and possibly some calendaring systems.

So, I've just finalized the master authentication server. It's done. Built. Finished. Kaput. The rest of our servers are still in various states of finality, and we have until September to lock them down. But right now, unified authentication is, for all intents and purposes (and with the exception of mail), working. And we couldn't be happier. The ultimate test will be, of course, letting users loose on this new infrastructure. I'm betting they'll like it almost as much as we do. At least the ones who know the old system. New users will be none the wiser. Ain't that always the way?

*Sigh*

Labels: ExternalNetworkUnification, Lab, MacOSX, NIX, Server, Systems