The Adventures of Systems Boy!

The last time we visited this topic, I thought we were done. Well, turns out I was wrong.

Things are, for the most part, working well now. Finally. We're running Tiger and we've managed to iron out the bulk of the problems. There is one issue which has persisted, however: the home account RAID.

To refresh, our network RAID, which is responsible for housing all our users' home account data, is made by a company called Panasas. Near as I can figure, we've got some experimental model, 'cause boy does it crash a lot. Which is not what you want in a home account server, by any means. After upgrading the Panasas OS awhile back, the crashing had stopped. But it was only temporary. Lately the crashing is back with a vengeance. Like every couple of days it goes down. And when it goes down, it goes down hard. Like physical-reset hard. Like pull-the-director-blace-and-wait hard. Like sit-and-wait-for-the-RAID-to-rebuild hard.

Again: Not what you want in a home account server.

So we've built a new one. Actually, we've swapped our backup and home account servers. See, awhile back we decided it would be prudent to have a backup of the home account server. Just something quick 'n' dirty. Just in case. This was built on a reasonably fast custom box with a RAID controller and a bunch of drives. It's a cheap solution, but it does what we need it to, and it does it well. And now we're using it as the new home account server. So far it's been completely stable. No crashes in a week-and-a-half. Keep in mind, this is a $3000 dollar machine, not a $10,000 network RAID. It's not that fast, but it's fast enough. And it's stable. By god it's stable.

And that's what you want in a home account server.

Moving to the new server — which, by the way, is a simple Linux box running Fedora Core 4 — has afforded us the opportunity to change — or, actually, revert — the way login happens on the Macs. In the latter half of las semester, we were using a login hook that called mount_nfs because of problems with how Mac OS X 10.4.2 handled our Panasas setup, which creates a separate volume (read: filesystem) for each user account. Since we're now just using a standard Linux box to share our home accounts, which are now just folders, we have the luxury of reverting to the original method of mounting user accounts that we used last year under Mac OS X 10.3. That is, the home account server is mounted at boot time in the appropriate directory using automount, rather than with mount_nfs at login. Switching back was pretty simple: Disable the login hook (by deleting root's com.apple.loginwindow.plist file), place a Startup Item that calls the new server in /Library/StartupItems, reboot and you're done, right? Well, not quite. There's one last thing you need to do before you can proceed. Seems that, even after doing all of the above, the login hook was still running. I could delete the scripts it called, but it would still run. Know why? This will blow your mind. Cache.

Yup. It turns out — and who would have ever suspected this, as it's so incredibly stupid — login hooks get cached somewhere in /Library/Caches and will continue to run until these caches are deleted. I'm sorry, but I just have to take a minute and say, that is fucked up. Why would such a thing need to be cached? I mean maybe there's a minimal speed boost from doing this. The problem is that now you have a system level behavior that's in cache, and these caches are fairly persistent. They don't seem to reset. And they don't seem to update. This is like if your browser only used cached pages to show you websites, and never compared the cache to files on the server. You'd never be able to see anything but stale data without going and clearing the browser cache. At least in a browser — 'cause let's face it, this does happen from time to time (but not very often) — there is always some mechanism for clearing caches — a button, a menu item, a preference. In Mac OS X there is no such beast. In fact, the only way to delete caches in Mac OSX is to go to one or all of the various Cache folders and delete them by hand. Which is what I did, and which is what finally stopped the login scripts from running.

If this isn't clear evidence that Mac OS X needs some much better cache management, I don't know what is.

In any case, we're now not only happily running Tiger in the lab, but we've effectively switched over to a new home account server as well. So far, so good. Knock wood and all that. Between the home account problems, the Tiger migration, and getting ready for our server migration, this has been one of the busiest semesters ever. Though I keep this site anonymous because I write about work, I just want to give a nod to all the people who've helped me with all of the above. I certainly have not been doing all of this alone (thank god) and they've been doing kick-ass work. And, though I can't mention them by name, I really appreciate them for it. At the dawn of a new semester, we've finally worked out all of our long-standing problems and can get down to more forward-looking projects.

So ends (I hope) the Tiger Lab Migration.

Labels: Lab, Systems, Tiger, TigerLabMigration

Take it as a symbol of my incompetence. Take it as a sign of the inherent difficulties in unifying any highly cross-platform environment. Take it as an indication that major OS updates in said environments are often far more difficult than you might have ever guessed. Take it for what you will. I'm just happy and relieved to report that, after many months of difficulties, we seem to be out of the woods: Our Tiger Lab Migration is complete.

(Note the sound of me frantically knocking wood.)

There are several things that have transpired -- things mostly out of my control -- that have made this migration, at last, complete. If you recall, there were two initial major problems from which we were suffering at the outset of the migration: 1) Mac OS X 10.4.2 was unable to write files across filesystems, which was impeding our use of certain software, particularly Final Cut Pro, which is used heavily in our lab; and 2) our Panasas brand home account RAID (on which we host our networked Mac home accounts via NFS) was crashing, almost daily at the end there. Problem number one was mitigated by certain workarounds I was able to implement using mount_nfs and loginhooks rather than the preferred automount method for mounting home accounts on our client workstations. Problem number two had both myself and the Panasas engineers scratching our heads.

Ironically, both problems are solved by system updates.

The inability of Mac OS X 10.4 to successfully write files across filesystems is cured, thankfully, with the latest update to the OS: 10.4.3. I have not yet implemented 10.4.3 lab-wide as yet, and so we're still running my mount_nfs workaround on the client systems, but on my admin machine I am testing the new update, and so far so good. Things behave properly again. I will also be running the update and the automount method on a test system on the floor for good measure, but I'm reasonably confident that this issue can, at last, be put to rest.

The home account server crashes also seem to have been cured by updating the Panasas RAID OS software. We upgraded on Friday, November 4th, 2005, and have not had a single crash since. This could be mere coincidence, but I'd be pretty surprised if that were the case. The crashing seems to have stopped, and I'd bet a fair chunk of cash that the update is the reason why this is so.

The best part of all this is that, for the past week, the lab has been quiet. People come in. They log on to the systems. They do their work. And everything functions properly, without incident, for the first time in months. The knocks on my door, the panicked moments, the hair-pulling-filled long nights and weekends have all subsided. My stress levels are slowly returning to what they once were, which is not to say normal, but rather normal for me. It is truly a thing of beauty.

There are two things a SysAdmin likes best in the world: a good challenge, and successfully overcoming said challenge. The Tiger Lab Migration has afforded me a hefty dose of both. And despite all the trauma, it's been an enlightening and illuminating experience from which I've learned a very great deal.

Labels: Lab, Systems, Tiger, TigerLabMigration

Well, when last we visited this issue, we were having problems saving, among other things, Final Cut Pro files. After working long and hard with the Panasas folks, I was confident I'd found a solution. And I had, but the solution has created new problems. I want to kill Apple for Tiger.

To refresh, the original problem was that Final Cut was trying to write files across filesystems, and the OS was choking on this operation. This was happening because our RAID exports its /home volume -- and this is what we mount -- but each user account is also an individual volume. FCP writes to the /home level, and then has to cross filesystems to save the file to the user's home account, itself a seperate volume. Our solution was, rather than mounting all of /home, to mount each user's home account individually. This required a whole bunch of scripting and launchd voodoo to really work the way we wanted, particularly with regards to adding new users. But we got it working, and it fixed the Final Cut problem. I implemented it, tested it, and emailed the community about the fix.

Unfortunately, I recently discovered that using this method causes some new, less ugly, but certainly annoying problems: the sidebar home account link no longer works, causing untold confusion among users who suddenly think their home account no longer exists (as per the alert message that appears when clicking this link); but most alarming, users can no longer open documents from their home accounts. Not only can users not open docs, but root via cron fails to open documents. This is a big problem, as our "Scratch" partition gets deleted every Friday, and the users rely on an alert message to warn them of the oncoming doom. This message no longer launches, and that's a real bummer.

So I was in the middle of drafting an email to the community informing them of the latest bugs and workarounds, when I was struck with extreme embarrassment and shame. How can I continue to expect users to work around these bugs when each week brings a new set? Sure, it's not my fault that Tiger is broken. It's not my fault that the Panasas is set up the way it is. But it is my responsibility to create a user environment that works seamlessly and properly for the user. So I resolved to do everything in my power to implement a solution. I stayed at work on Friday night until 2:30 AM, learning much, but left with no solution in hand.

What I learned on Friday, essentially, is that automount in Tiger is totally fucked. I already knew that NFS in Tiger was fucked in that it can't cross filesystems. But it turns out there have been some major changes to the way automount is handled in the GUI, and thus, for all intents and purposes -- or at least for our intents and purposes -- it is indeed fucked. Hard. The inability to follow sidebar references is a direct result of automount's new Finder behavior: automount mounts in the Finder are now invisible! Unless the path is explicitly called, the user accounts, mounted as they are, are quite invisible, both to the user, and apparently to the Finder's sidebar. I also discovered that the Finder's inability to launch files from the user's home account has something to do with automount, or at least how automount is implemented in the GUI. This behavior only exhibits itself from these invisible NFS mounts; it goes away if we mount the user's home account in /home, as we used to do. And it goes away if we use a different command to mount user homes.

Enter mount_nfs.

The mount_nfs command is used to directly and statically mount NFS exports, and it has its own set of peculiarities. First and foremost, mount_nfs grafts mounts to existing folders, which means that mount_nfs requires a folder named for the user who wants to log in. Secondly, and equally important in our scenario, mount_nfs magically mounts the export in the Finder. Or, rather, shows the mount in the Finder (and this has actually, probably more to do with how the Finder translates mounts requested by nfs_mount). Lastly, since the mounts are static, they remain mounted and appear in the Finder until they are explicitly unmounted. Mounting via mount_nfs is equivalent to mounting nfs exports in the Finder with command-k. It also seems to work, you may notice, completely opposite to automount.

To do what we've been doing -- which is to automount home accounts at boot -- using mount_nfs instead would require us to mount every home account at boot on every machine, and those 200+ mounts would be present in the Finder at all times. This simply would not work. So, mount_nfs at boot: not good. You thinking what I'm thinking?

Enter loginhooks.

To use mount_nfs at boot would be disastrous. What we need in this case is something that will dynamically mount and unmount users' home accounts at login and logout respectively. And that's just what loginhooks and logouthooks do, respectively. Getting loginhooks to work in Tiger was again an exercise in frustration, but this time it was due to my own poor understanding of the technology. There is a great reference on login hooks at Mike Bombich's site, and a decent one at Apple's Knowledge Base as well. Between these two resources, I was finally able to cobble together a solution over the weekend which I think will work.

Briefly, there are three things you need to do to implement loginhooks, and some info you need to know.

The info first:
1) Scripts called by loginhooks run as root (good)
2) They run before login to the GUI takes place, and after authentication (also good)
3) The user requesting login can be represented in your script by the variable $1 (excellent!)

What to do:
1) Write a script to be executed at login, make it executable, put it somewhere accessible
2) Run this command:
sudo defaults write com.apple.loginwindow LoginHook /path/to/loginScript
3) Test the script! This is imperative! You can test it as any user you want by running it thusly:
sudo /path/to/script User

For "User," specify a user who would actually log in, and who is available to the system. "User" here will be interpreted in your script with the $1 variable, just as it will at login. If it works in your test, it should work in practice as a loginhook.

Setting up a logouthook works the same way, except you run the command:
sudo defaults write com.apple.loginwindow LogoutHook /path/to/logoutScript

And, BTW, to check that the loginhook (or logouthook, or both) has been successfully added, you should see your script(s) listed, when running this command:
sudo defaults read com.apple.loginwindow

Finally, to disable, login(logout)hooks run:
sudo defaults delete com.apple.loginwindow LoginHook
and:
sudo defaults write com.apple.loginwindow LogoutHook

So, what I have now are two scripts. The login one creates a folder in /home named for the user who's logging in, and then uses mount_nfs to requset the user's home account from the server and mount it in this folder. The logout one forcibly unmounts the user's home account. And that is all.

I've only tested this at home, but it seems to work brilliantly here, and is fast even over a wireless connection. I will be trying it at work on Monday morning and will post back with my results, success or failure.

Wish me luck.

UPDATE 1:
So far, so good. There were some snags, though. I came in a little early this morning (in hopes of starting before the lab became overrun with students) to implement and test my loginhook plan, only to find the home account server had crashed. So, I had to spend the first hour restarting and troubleshooting the Panasas. By the time I'd finished, the lab was filling up. So I had to do everything piecemeal, and it took a bit longer then I'd hoped. That was snag one.

The next snag was that, after one user logged in, the next user would be locked out of his/her home account. This turned out to be a simple matter of setting permissions (on the /home directory) via the loginhook in a manner in which this did not happen.

There are some advantages and disadvantages to this new mounting method. The greatest benefit is that any changes to the mount method are done via very simple shell scripts, and changes don't require a reboot to take effect. Just change the script and you're done. Another big plus it that, if the home account server goes down, the machines aren't really affected to the extent they once were. If no one's logged in, there's no server access happening, so the machines are fine. Only if someone's logged in is it a problem, but then, that's always a problem. It occurs to me, though, that I may want to add a -hard option to my mount script sometime soon. Another advantage is the fact that, since only individual user accounts are mounted, and not the entire home account server, a command like sudo rm -rf /home can't wipe out the entire server as it could before. The final advantage is that reboots are now much faster, since nothing happens at boot time.

The disadvantages are twofold, and solvable but minor: First, since the entire home account server is never mounted now, users no longer have easy access to each other's home accounts. To access another user's home account, they now must connect to the home account server via the Finder's "Connect to Server..." command (or command-k). Secondly, any user who wants to ssh to one of the other Macs, say, from a laptop, will not have access to his home account since the mount only occurs when logging in to the GUI. These issues are minor, and can be solved by simply mounting the home account server somewhere other than /home, but I think we'll try and live without it for now, as I kind of like not having the whole RAID mounted.

Anyway, no complaints thus far. I'll be convinced this is a go after a week or so without problems. But I'm cautiously optimistic at this point.

UPDATE 2:
Day two, and all's quiet on the Western front, as it were. I'm obsessively monitoring the situation, but it's looking very good at this point. Fingers crossed.

Labels: Lab, Systems, Tiger, TigerLabMigration

Boy have we got troubles. Sure I tested everything. Sure I did everything I could to make sure this upgrade went smoothly. But does that ever matter? No. It doesn't.

Everything tested out fine. Tiger was able, from day one, to mount our NFS home account server -- a network RAID known as the Panasas. It was able to read files, to write files. It seemed fine. But we'd had our fair share of troubles with the Panasas, even in the beginning, even in Panther. Certain Macromedia products -- actually, Flash MX 2004, to be specific -- had certain functional disabilities. In particular, when any user was logged onto the Mac as a network user whose home account was located on the Panasas, he would find himself unable to publish HTML previews from within the Flash MX 2004 application. The error message was something along the lines that Flash was unable to read the HTML templates, which live in the user's home account. Oddly, on first launch, Flash could write the templates, but for some reason, it could not read them for the preview. If you have any Flash knowledge whatsoever (which I don't -- I was informed by student experts) this renders Flash essentially useless. The problem did not, however, exhibit itself with network home accounts mounted on Apple shares, neither via AFP nor NFS. So the workaround was to create a special account for Flash developers, which they could log into when doing Flash work, and which was mounted directly on the MacServer and shared via AFP. Hence was born our FlashDev account, which has worked fine all year.

Jump ahead to the present: we've upgraded to Tiger. It seems to have gone fine. Suddenly, certain apps begin acting strangely. Illustrator can't save files. Word also has trouble saving files, and behaves erratically, complaining of "permissions problems" and the inability to write temp files to a "network disk." Final Cut, too, begins acting flakey. Shit. It's like one of these sci-fi-horror flicks where the brain transfer experiment seem to have gone perfectly, but then, gradually, the patient begins acting... Different somehow...

And then all Hell breaks loose.

Tracking down all the problems, I finally came to the conclusion that a number of applications are unable to read their preference files in Tiger when those files reside on the Panasas. This perhaps causes a chain reaction that makes these apps unable to properly write or save files: FCP writes 0 byte files; Illustrator CS just can't save your document. Oddly, Illustrator CS2 doesn't seem to suffer from this problem, and all these apps (except, of course, Flash) seem to work properly from within the Panther environment.

Seems to me like a nasty reaction between Tiger and the Panasas home account server.

I've done some testing, and I plan to do more. So far, I've tried booting into a Panther client and found that the problem is greatly reduced in Panther (though it always existed to some extent, particularly, again, with Flash) and exacerbated in Tiger. I've also tried resharing the Panasas NFS export via AFP from the MacServer. I was fairly sure this would provide a reasonable, if temporary, workaround, but it didn't. I want to try it again, just to be sure, but if resharing via AFP does not provide a cure, it would indicate that the fault lies somewhere with the Panasas. The other possible culprit is Tiger's implementation of NFS. To test this, I am building a more generic (non-Apple, non-Panasas, non-proprietary) NFS server to test from, on a Linux install on my new Dell laptop.

So, best laid plans, and all that. What a frickin' mess.

I'll keep you posted as I learn more.

UPDATE 1:
I built a Linux install for the express purpose of testing networked Mac home accounts on a non-proprietary (non-Mac, non-Panasas) NFS mount. When running a Mac home account from this mount, the Final Cut problem disappeared, and the application acted normally. The same was true for Microsoft Word. Illustrator CS still behaved badly, though, as did Flash MX 2004. It's clear now that these problems result both from Tiger's implementation of NFS as well as Panasas's. So, the breakdown goes something like this:

• The Flash problem is probably Flash/NFS-specific (though possibly Mac-specific as well)
• The Illustrator problem is Tiger/NFS-specific
• The Final Cut and Word problems are Tiger/Panasas-specific

None of this makes my job any easier. It kind of sucks to realize that there is no magic bullet, because that means that there probably is no single fix. Still, it's useful to know.

UPDATE 2:
The issue with Final Cut is not a matter of the application being unable to read files; it's a matter of it being unable to write them. Essentially, FCP cannot write a preference file. Launching with no preference file, FCP creates a 0 byte preference file in the user's home account. Also, if you try to save the FCP project file to the NFS share, it will be a 0 byte file and unopenable. Saving the FCP project file to a local volume results in a perfectly usable FCP project file. The complete inability to write files seems to be unique to Final Cut. Other applications (Illustrator, Flash) seem to have trouble reading files, but writing them seems to work fine (which is why I assumed this was the case in FCP). Word (now updated to 11.2) also has trouble writing files, but only when saving an open, modified document. That is, if you create a new document in Word, and hit "Save," it saves just fine. If you keep the document open, make changes, then hit "Save" again, it complains that it can't write to the shared disk. Hit "Save" again, and Word will create a new document and use the first string of characters in the document as the name. "Save As..." however, works as expected.

UPDATE 3:
It occurs to me (through the suggestion of other, smarter folks) that maybe this is not completely an NFS problem, particularly in the case of the apps that have problems writing files (Word and FCP). Those apps work just fine when we mount our homes on a generic NFS export. So it's possible that the write problems we're having are filesystem-related, not NFS-related. This theory holds a lot of water considering the fact that the Panasas uses its own, proprietary file system.

UPDATE 4:
I've been in contact with the Panasas folks, and I'm working with them in trying to determine the problem. So far, I've sent them tcpdump results from both the Panasas and the Mac while FCP tries to save a file. I'm told that there are a "massive number of 'file not found' type errors" from the dump. That can't be good. I've also sent them a few file listings from the home account of the user in question. Meanwhile, I'm wondering if the next Tiger update will do anything to correct these issues, or if upgrading the Panasas shelf would help, while at the same time, trying to come up with a reasonable workaround, in case Panasas is unable to reslove the issue. The only thing I can think of is to move the Mac home accounts to another server. Which would suck hard. Two steps backward.

UPDATE 5:
Still in constant contact with Panasas. I'm sending them log files now, and planning to do a ktrace, which will be a new experience for me. Also, I'm seeing this issue occur with other applications now. In particular, Macromedia Director MX 2004 is unable to write files to the Panasas mount. This is particularly odd because under Panther that application had a completely different problem: it just wouldn't launch. So I'm currently dealing with three seperate incident reports: one for the problem noted here, one for a blade crash that happened a couple of weeks ago, and one for the original Flash problem from our Panther days. It's a pain. Fortunately, the Panasas people are very nice, professional, and helpful.

UPDATE 6:
So, this does seem to be a filesystem issue at its core. Sort of. After much work with the Panasas guy, we've narrowed down the problem considerably.

Mac applications write temporary files -- files that are in use but that haven't yet been saved -- to various locations depending on the app. Final Cut writes its temporary files to a folder called .TemporaryItems at the top of whatever volume the user is working from. So in our model, the top volume is /home, and the sboy home account is inside /home. The whole thing looks like this: /home/sboy. Final Cut wants to write its temp files in /home/.TemporaryItems. And it is able to do this. If I look in /home/.TemporaryItems, I find all the files I've been unable to save to my user account, perfectly intact. So, temporary files go in /home/.TemporaryItems, and saved files in /home/sboy. Now, here's the grind: The Panasas setup requires that each home account be a seperate volume. So, /home is one big volume, and /sboy is another volume inside /home. This means that when Final Cut tries to move the temporary files from the .TemporaryItems folder, which is located on the volume /home, to the sboy user account, which is another volume at /home/sboy, it is moving the files across filesystems. And we get an error.

Interestingly, moving files across filesystems when Panasas is mounted on a Linux box produces the same error, but Linux does something to compensate for this problem, and is then able to save the file. But the error still gets produced. It's just that Mac OS X 10.4.2 (this did not happen in Panther) does not have the mechanism to correct for this error. It gives up and leaves the file in its original location: the /home/.TemporaryItems directory.

Also, this explains why everything works fine when I mount the user's home account on my Linux NFS export. In that case, everything is on the same volume, and there is no moving of files across filesystems. It's got nothing to do with NFS at all.

I'm curious to know why this doesn't happen in Panther. Is it possible that Panther has the same error correction function we see on Linux? What about earlier versions of 10.4? Do they have this mechanism? Will 10.4.3 have it? I surely hope so. I'll be looking into who I should contact at Apple about all of this.

Incidentally, the Panasas guys have figured all this out by examining ktrace and tcpdump files made while the program ran and the error occured. I did not figure any of this out on my own. All I did was run the commands and upload the files. Still, it's a nifty bit of sleuthing. I'm learing a lot.

UPDATE 7:
So, after mulling this over for a while, a rather obvious workaround occured to me. Currently, we mount the entire Panasas volume on our systems. This volume includes all home accounts within it, as well as the .TemporaryItems folder at its top level. Each user's home account on Panasas is a separate volume, and so is the top-level volume that contains .TemporaryItems, so saving files must cross filesystems (from the .TemporaryItems folder on /home to the user's folder). The workaround is to mount each user's home account individually. Doing so causes FCP to treat the user's home account as the active volume, and to use the .TemporaryItems folder in said home account. In this scenario, no filesystem boundaries are crossed, and no error occurs. I tried it, and it works. FCP (and Word, incidentally) can now both save files to the user's home account normally, and application preference stick. Final Cut is again usable.

Hallelujah!

I'm testing this on a couple machines before I go and announce it to the community and put it on all the Macs, just to make sure it works properly. But it's looking good. I'm glad to know that the past two weeks I've spent on this haven't been in vain.

Labels: Lab, Systems, Tiger, TigerLabMigration

Well, we're getting close. We're down to last, wiggly-niggly little tidbits. Here's where we've been in the last few weeks:

The Master System
We have completed a very nice build of Tiger with most of the software we need on all machines, though we're lacking all the most recent Adobe stuff due to not having received the most recent Adobe stuff. 'S'okay. We'll make do with CS1 for now. Our system is all loaded up with our custom scripts, and a new, beefier home account mount script. And it's got a nice, big, 75GB system partition. Why, it's just swell.

We've used this system to clone to all our other lab machines. We boot these other machines in firewire target disk mode, and then we use Disk Utility to restore the Master System to the new machine's system partition. Or, we wanted to use Disk Utility. Turned out that our first Disk Utility-cloned system had some problems due to -- yep, you betcha -- Tiger bugs. The problem was that Disk Utility wasn't blessing the new system partition, so when we'd boot the new machine, we'd get the flashing question mark of doom. So we'd have to then boot off the Tiger install disk, and set the new system partition on the newly built machine to be the startup disk. I decided this was a pain, and that it would be much easier and more instructional to do our clones with a tiny, tiny shell script that calls ASR (Apple Software Restore, via the asr command).

#!/bin/bash

## Script to Clone Local Volumes ##
## systemsboy - Aug 18, 2005 ##

clear

## Define Variables ##

echo "
Please enter the path to the SOURCE volume or ASR-ready disk image...
(Example: /Volumes/MyVolume)"
read SOURCE

echo "
Please enter the path to the TARGET volume...
(Example: /Volumes/MyVolume)"
read TARGET

## ASR Portion of Script ##
echo "
Building Clone... Please wait...
________________________________
"
sudo asr -source "$SOURCE" -target "$TARGET" -erase

exit 0

This did the trick indeed, and was as easy to use, if not as purdy, as the Disk Utility application. This method only yielded one problem, and that was that all the invisible folders -- /var /etc /private and the like -- were not made invisible on our cloned systems. This is another Tiger bug, but one that was fixed in 10.4.2's version of Disk Utility. Apparently not fixed in ASR from the command-line, however. This problem yielded both an intersting solution, and some interesting information. Back in the day, these files were made invisible to the Finder by the magic of the .hidden file. The .hidden file was simply a list of files or folders at the top level of the root drive that should be treated as invisible by the Finder. Want to make something invisible on /? Just add it to the .hidden file and restart the Finder. (Ironically, the .hidden file was also at the root level of the system, but was rendered invisible by the fact that its name began with a period.) Nowadays, however -- in Tiger, that is -- this is no longer that case. Files on root are now made invisible, according to this Apple KBase article, by setting certain file attributes. I'll tell you, I have yet to figure out what those attributes are, or how to modify them, but when I do I'll shout. In the interim, I was forced to locate the "SetHidden" program on my original Tiger install DVD. After insering the DVD, it can be found at:

/Volumes/Mac\ OS\ X\ Install\ DVD/System/Installation/Packages/OSInstall.mpkg/Contents/Resources/

I've now copied this file to a folder called SetFile, which I've placed in the Applications folder for future use. Also, I needed the hidden_MacOS9, which can be found, and should be placed, in the same folder as the SetHidden command. So I now have a folder in my Applications folder, called SetHidden, in which the SetHidden command and the hidden_MacOS9 files reside, installed on my master, and I've added the commands:

cd /Applications/SetHidden
sudo SetHidden /Path/To/My/Clone/System hidden_MacOS9

to my shell script, and voila! Clone script complete!

#!/bin/bash

## Script to Clone Local Volumes ##
## systemsboy - Aug 18, 2005 ##

clear

## Define Variables ##

echo "
Please enter the path to the SOURCE volume or ASR-ready disk image...
(Example: /Volumes/MyVolume)"
read SOURCE

echo "
Please enter the path to the TARGET volume...
(Example: /Volumes/MyVolume)"
read TARGET

## ASR Portion of Script ##
echo "
Building Clone... Please wait...
________________________________
"
sudo asr -source "$SOURCE" -target "$TARGET" -erase

## Set Hidden Files (Tiger Bug Workaround) ##
echo "
Mac OS X 10.4 does not properly set hidden files.
We will now use the SetHidden command to rectify the problem.
For this step, the SetHidden folder must be present in /Applications.
---------------------------------
"
cd /Applications/SetHidden
sudo ./SetHidden "$TARGET" hidden_MacOS9

exit 0

Cloning all 14 systems on the floor took my Lab Assistants a few short hours.

Spotlight
I mentioned some concerns regarding Spotlight and the particular way in which users use our lab. After some mucking around, and testing of various methods of disabling Spotlight, both forced and optional, I've decided to do nothing. I do this kind of thing all the time. Worry and worry about something, and then, finally, throw caution to the wind and do nothing about it. It's my way. It's part of what makes me so gosh-darn lovable. But I do have two things behind that decision: a rationale, and a backup plan.

The first problem I'd anticipated was that Spotlight would begin indexing user home accounts over the network all at once as soon as everyone logged in, and that this would cause problems with network performance and incomplete indexes. Upon further reflection (read: obsessing), I decided that that Spotlight's network resource usgae would probably be low, and that it would probably avoid incomplete indexes by either continuing in the background after logout, or by resuming at next login. Also, since home accounts have quotas, the largest of which is 7 GB, indexing shouldn't take too long, and any problems should be quickly mitigated.

The second possible problem I'd worried about was that Spotlight's insistance on indexing firewire drives would take forever to yield complete indexes on large drives -- a process that would likely be interrupted at logout before it was complete -- leaving many users with partially indexed drives. In my tests, however, Spotlight picked up right where it left of with drives that were partially indexed when ejected and then re-mounted, so I'm not convinced this will even be a problem. I also worried that Spotlight's firewire drive indexing would cause performance problems that would incapacitate users who were working on video. This may yet be the case. We'll know soon enough.

I came up with a number of possible solutions to these problems, all messy, all imperfect. Ultimately, I decided that the best option was to wait and see what problems arose, and then tackle those problems with the most appropriate of the solutions I devised. Who knows? Maybe it will all work out great without any intervention from me. And that'd be just ducky.

Network
We have an intersting and complex network. It consists of Mac, Linux, and Windows machines, a network RAID, and web and mail storage. And all of this is accessible, in one way or another, via the network. The problem comes when you try to explain all this to new (or sometimes even old) students. It's not an easy picture to paint. And overwhelmed first years usually don't quite catch on for some time. But as we design our network, and build our systems, we're very concerned about how all that is presented to the users. And we try to make it as simple and as understandable as possible, which is, after all, the Mac way, and is why people love the Mac. So let me just say, right here, right now, what the fuck is up with the "Network" view? That piece of shit gets worse with every OS revision. And no one seems to care. I can't find diddly about it on the forums.

So here's the background on this outburst. Back in the Panther years, the Network view offered up a pretty nifty way to present a great deal of our network -- the shared drives of all platforms -- to our Mac users. The coolest part was that it looked much like what you saw on Windows, which simplified things greatly by presenting a consisten network view across platforms. There were a couple things that made this possible. One was SMB workgroup names. SMB does a great job of not only sharing files, but broadcasting itself as a service on a network. Setting up computers under SMB with a workgroup name based on the platform of the given machine got all our Mac, Linux and Windows computers grouped appropriately on the network. And Mac can read those groups and smartly creates folders in the Network directory named after those groups and populated with the systems that belong to them. So in my Network browser, there'd be a folder called "Linux," and all the Linux computers would be in there. Same for Windows. Here's the catch: Our Macs share via SMB to Windows, but we want them to share via AFP to other Macs. Personal File Sharing, which uses AFP, does not, by default broadcast any kind of grouping. If you turn on AFP, you just broadcast the service, but you can't specify it's appearance or where on the client the shares will appear. Computers broadcast via AFP in Panther just show up in the "local" folder, not in a "Macs" folder like their SMB sharing counterparts. There was, however, a sneaky way to define this in Panther, by editing a little file called /etc/slpsa.conf, and thereby magically creating what are known as SLP scopes. In fact, SLP scopes are just that -- a way to group a particular service from a particualr device into logical groups. So we scoped our Macs to the "Macs" scope and they showed up in our "Macs" group in /Network, and all was well in the universe. If you're unbearably astute, you'll probably be wondering why, though, our Macs, when connecting to other Macs via the "Macs" scope in the Network folder, didn't connect to said Macs via SMB rather than AFP. And here was something amazingly cool in Panther. Panther smartly dealt with the folder full of Macs in our Network view. So, when it saw these other Macs broadcasting and sharing over two protocols -- AFP and SMB -- it intellegently chose the best one, which for a Mac was AFP, of course. That was the thing of beauty that allowed us to present the same view of the entire, heterogeneous network the same way on all platforms while still allowing us to connect with different protocols. It was frickin', frackin', goddamn sweet, is what it was. Brilliant. And now it's gone.

Not only is this intelligent decision making gone from Tiger, at least in the latest incarnation (10.4.2 as of this writing), but so too is the ability to define SLP scopes. So, this means two bad things: 1) I can no longer define specific locations for my AFP-shared Macs, and 2) any connection made via the "Macs" folder in the Network folder (which, remember, is now only defined by SMB, not by SLP) connects via SMB; my Macs connect to each other via the Windows file sharing protocol, which is a big fat problem.

So, I'm currently trying to devise a way to scope the Mac shares into some sort of logical grouping that will allow me to present those shares in a palatable way. But it ain't easy. I know you can do this with Tiger Server, and this is probably why they've crippled it on the client, but I don't a Tiger Server yet, and I'm hesitant to build one this close to the beginning of the semester. So we shall see...

The migration is nearly complete. Time permitting, I would abasolutely love to build a fresh new Tiger Server. I'll probably give it a shot in this last week before classes begin, but I'll be sure and keep a clone of the existing Panther Server in case things go awry. Beyond that, and the aforementioned wiggly-niggly, we're done. We're upgraded, on the workstations anyway. And it is good.

There will, most likely, be problems that arise once the students come in and start using the new systems. I will share those in a later post. Until then...

Labels: Lab, Scripts, Systems, Tiger, TigerLabMigration

So this is the part of this epic in which I build what I call the "Base Config" or "BC." The idea behind the BC is simple, really: Build a machine that's got it all (well, almost all), from which all subsequent machines in the lab can be cloned. Building the BC is always a little scary, because any mistake I make on the BC will be propegated to about 25 machines, and consequently will have to be corrected on said 25 machines. So I've got to be careful and thorough in my planning.

Essentially, all my machines are the same, or at least share the same core: the latest and/or greatest version of Mac OS X, major applications from Adobe, Macromedia, Microsoft, and of course Apple, and some smaller applications here and there, mostly utilities and drivers or things like Suitcase. These things go on every Mac in the lab. So they go into the BC Mac as well.

In addition to the OS and the applications, there are some admin things that need to get done: We have some custom scripts and dock items we like to put on the Macs, as well as a Startup Item to mount our home account server via NFS. And, of course, Directory Access must be configured to get authentication, and whatever other services we set up, from our Macserver. Then each preference pane in System Preferences should get configured the way we want. Finally, we add a few things to /etc/hosts and there are a couple cron jobs that need to get setup. And I believe that's it.

And, like I said, I hope that's it, because if it's not -- if I've missed anything -- I'll be paying for it later.

Here's where lists start to come in real handy:

• Mac OS X 10.4.2

1. Install OS
2. Install all Software Updates
   

• Local User Accounts

1. Me (admin)
   
2. Lab Assistant (admin)
3. Student (generic non-admin)
   

• System Preferences

1. Configure All
   

• Adobe

1. Photoshop
2. Illustrator
3. InDesign
4. Acrobat
5. AfterEffects

• Apple

1. XCode
2. Final Cut Pro Suite (FCP, DVDSP, Motion)

• Macromedia

1. Director
2. Studio

• Microsoft

1. Office 2004
   

• Other Software

1. Stuffit
2. Suitcase
3. USB Serial Drivers
4. WACOM Drivers
5. KeyServer Software
   

• Admin Junk

1. Configure Directory Access to authenticate against MacServer
2. Mount Home Account Startup Item
3. Admin Scripts (local delete, quota alert)
4. Add servers to /etc/hosts
5. Add cron jobs (local delete)
6. Spotlight Disable Script (so that home accounts do not get indexed)
   
7. Application Menu
   

So, that should do it. I'll build this, start testing it, and add anything to the list I forgot. But that's pretty much it. Once this is built and working well, it will be time for the trial by fire. We'll start cloning this machine to the other workstations. This year will be extra special fun, because not only will we be cloning these, we'll also be wiping and repartitioning the internal drives of all our machines. Fortunately I have lots of firewire cables, and very capable and energetic Lab Assistants who are ready, willing, and able (and paid, for that matter) to help me out with all this.

And one last side note: As I build this machine, just for fun, I may create disk images along the way of slightly leaner builds than the final. Like a build with just the OS, then one with just the commercial apps, then one with the drivers, and finally one with all the fixin's. This way I have the various stages available to me in case I need to build, say, staff machines, from a leaner base system, or in case I screw something up and need to go back a step or two, I won't have to start completely from scratch.

So that's the plan. I'll let you know how it goes.

UPDATE 1:
I've just finished the first stage: installing and updating the system software. The OS is at 10.4 2 and all Software Updates have been applied. I have also configured my account, and the other local accounts, and configured all the System Preferences. I have created a disk image of this install, called SysAppsBC-BaseOS.dmg, and scanned it for ASR.

Labels: Lab, Systems, Tiger, TigerLabMigration

I've been building my Master Tiger system. Actually, it's built, and I've been testing it. And I was going to make it my Radmind Master Client, but I've since scrapped my Radmind plans altogether. So fine. It's now just a matter of uninstalling Radmind from the Master and testing it and making sure all is well. Well, all is not well. Somewhere along the line, Apple File Sharing broke. I don't know exactly how or why. All I know is that, yesterday, after uninstalling Radmind, I tried connecting to the Master from my admin machine via AFP. The share mounted for a few minutes, then my Finder beachballed and I got a new error message alert. I'd never seen this one before:




Pressing the "Disconnect" button does just what you'd think: It disconnects you from the now defunct share.

So, looking into this a bit more, I've discovered that the AppleFileServer process on the Master is crashing whenever I try to connect to it via AFP. After the crash there is copius and completely useless output in the AppleFileService.crash.log, and Personal File Sharing is off in the Sharing System Preference pane. I can connect in the opposite direction -- from the Master to my admin box. File Sharing on the Master, however, seems hopelessly broken.

I'm not sure what to do.

I figure I have three options: 1) I can spend forever and a day trying to figure out what went wrong and maybe fix it, 2) I can wait for the much-anticipated 10.4.2 update and hope that whatever got broken on this machine gets overwritten with fresh new copies from the update, or 3) I can wipe and reinstall. Since it's likely I'll have no luck figuring out the problem, and since I'm not really the wait-and-hope type, I think I'll go with option 3.

Time to build another system.

I hate Tiger.

Labels: Lab, Systems, Tiger, TigerLabMigration

Here's an interesting little gotcha. Not sure if it's a good thing or a bad thing yet.

Probably bad.

In our lab, home accounts are on a server. Or, more accurately, they're on an NFS mount that is shared from a network RAID. The way this works is fairly simple, but kind of tricky. The Macserver handles the authentication for network users who log into client workstations. The clients always have the NFS RAID mounted at /home. When users log in, Macserver specifies their home accounts as /home/username. To make sure the RAID is always mounted, we use a little startup item that's just a very simple automount script to call the RAID and mount it in /home.

There's one other little thing that's weird about our setup, and that's the way the RAID is configured. Our RAID is a very nice, but proprietary system made by a company called Panassas. The way user accounts are created on the RAID is unique, and I don't fully understand it, as I did not set the RAID up, nor do I maintain it. But essentially, from my understanding, each user's home account on the RAID is a separate partition.

Does anyone see the problem here?

Well, I won't keep you in suspense. If you haven't figured it out, here's the problem -- and the more I think about it, the more I realize that it is a problem and not a boon: Whenever a user logs in, a new partition is mounted via NFS. And guess what happens then. You guessed it (or maybe you didn't): Spotlight starts indexing.

Holy fucking shit.

I have 207 users currently on the RAID. They have quotas between 2 and 7 gigs. And they're mostly completely and totally unaware of Spotlight and its idiosyncrasies. If one of them logs in and then, say, shuts down the machine (it will happen, trust me), the Spotlight index will get hosed and the machine will most assuredly begin acting flakey. Or how about this: What if a user logs in, checks his email, then logs out? Then what if another user logs in, does same, and logs out? What if five users come along and do this? Now we've got spotlight indexing five different network mounts on the same machine. What if that machine then gets rebooted before indexing is complete? I shudder to think. Or, what if a user logs in to a machine, indexing begins, she logs out and then logs into another machine? Now Spotlight is indexing the same mount point -- accessing the same database -- from two different machines. I'll say it again: Holy fucking shit.

This is a recipe for disaster.

Fortunately, I am an expert in the various methods for turning off Spolight. And that's what we'll have to do: turn off Spotlight for all 207 mount points. This isn't really that big a deal. One simple command should do it (I hope). But it gets me thinking about all the various other Spotlight related problems we're bound to encounter. For instance, our users do a lot of video, and they're encouraged to use firewire drives for this. Well, firewire drives are indexed as soon as they're mounted. I have no way to change this. What happens if indexing on a firewire drive is stopped (i.e. the user unmounts his/her drive) before it's complete? Now we have a hosed index on the user's firewire drive. Next machine he/she goes to will try to index the drive again, possibly completing the index, possibly not. And, during the indexing period, will performance drop to levels that do not permit video editing? I just don't know. But if they do, it's going to be a real problem. And just how do you educate 200+ users about this? It's way over most people's heads. This is a technology that is supposed to "just work." Unfortunately, in a multi-user, networked environment like ours, my worry is that it will "just break."

I'm feeling very hesitant about this migration. It wouldn't be the first time Apple's plans for the home user have come at the expense of the networked lab user. They often seem to forget about us, even though, in many ways, it's this sort of environment for which OS X is so great. Ironic. But if you think about it, one of the greatest features of Tiger -- Spotlight -- is completely useless in a networked environment. In fact, Spotlight is not even supposed to run on networked volumes. (Why it does on ours, I do not know, though I suspect it's because we're using NFS.) But the firewire thing is really disturbing, and I think really underscores the need for significantly more control over the behavior of Spotlight. If, in the Spotlight Preferences, there were a checkbox for "Disable Spotlight Indexing on External Drives," I'd be the happiest man alive right now.

As it is, I'm just plain worried.

UPDATE:
Another thought occurs to me: Okay, so I disable Spotlight on all 207 accounts. Well, what happens when we create a new account? Spotlight needs to be turned off for that account too. So basically, what this amounts to now, is a script that gets run at least every time a new user account is created -- possibly at every login, just to be safe -- that disables Spotlight on all mounted home accounts.

Oh joy.

Labels: Lab, Systems, Tiger, TigerLabMigration

Okay. So, Tiger client is working. Moreover, Tiger client seems to work with my Panther Mac Server. And I have a backup disk image of my working Tiger install, should anything go amiss. Time to start setting up Radmind.


The Logic:I have about thirty Mac systems to maintain in a lab set up for art students doing all manner of computer-based art, including: web design, graphics, video, audio, interactive authoring (from screen-based to installation art), and some 3D. Certain software -- like the operating system, for instance, is installed on all the machines. Certain software -- like Max/MSP, for which we have only so many licenses -- is installed only on select machines. Also, some systems are workstations used by students for the creation of their work, but there are also staff machines which serve vastly different purposes and are set up very differently. This means we have multiple hardware/software configrations for the various systems in our department. Keeping these machines up to date can be very challenging. Not only do we need to keep tabs on which systems have which software, we also need to keep tabs on which systems have been recently updated and which ones are in need of being updated. In the past, this has meant keeping a database of system configurations, logging into or polling (via ARD or some such utility) systems to see which ones need updates, and, when updates are required, personally sitting at machines and running software updates by hand. This process is tedious, inefficient, and more importantly, quite error-prone: Do one thing differntly on one machine, and you've suddenly introduced inconsistencies throughout the lab. And with no way to track them, or even revert them should the need arise.

Clearly what is needed is a centralized system for software and OS update management and reversion, whereby changes to be made to workstations on the lab floor can be applied to a single system, tested, and then propagated to the appropriate systems during off-hours or scheduled maintenance times. Radmind is such a solution. And, miraculously, it's completely free.

Wow.


The Goal:We have several different configurations of Mac on the lab floor, and in the various staff offices. Before we start, let's outline them: We have Basic Workstations (BWs) with a basic (though still quite large) set of software; we have what I'll call Max Workstations (MW), which are essentially the same as the BWs, but with a Max/MSP added; we have Physical Computing Workstations (PW), which have the Max/MSP config plus certain drivers required for programming Basic Stamp and the like; we then have Staff Workstations (SWs), which have a leaner software set overall, but which also have software not generally found on the public workstations; next, we have Audio Worksations (AWs), which have the basic set plus -- you gussed it -- audio software and drivers; and finally we have Video Workstations (VWs) that are set up like the Basics, but with a few video do-dads to boot.

A quick note about the Audio and Video workstations: I share maintenance of these machines with our A/V SysAdmin. Essentially, he manages them, but I provide him with a baseline OS install (by way of some sort of cloned image) and advise him with regards to OS updates and the like; he installs, configures, manages and updates any audio- or video-specific software not found in my base config. This area of the lab will be tricky to control with Radmind, particularly the Audio Stations as they often require certain hardware to be available for the software to be installed. Also, since our A/V guy handles upadates to those systems, and since I do not (and this is a good thing), using my Radmind system for A/V updates might prove tricky. I will save the A/V systems for last, and figure out how best to handle them later. Fortunately, Radmind allows for this sort of gradual implementation. Ultimately, though, I may leave them out of my Radmind setup.

The ultimate goal will be to set up one alpha workstation that has everything required for every configuration. Then Radmind can be used to create subsets of this uber-station (henceforth referred to as the Master Client or "MC"), for propagation to workstations with less than the maximum of software installed. The MC will be built around the configuration of the Physical Computing Workstations, as those systems have all the software needed on any other system.


The Process:The first step is to set up the Radmind server, which will be my admin box. That's really easy, and is done. It's simply a matter of downloading the Radmind software packages, and then running the Radmind Assistant application. These can be found here. In the Radmind Assistant I just set my system up as a server. That's all. Oh, and I made sure to set it to use Bonjour for discovery. This makes server discovery from the client a breeze. The client simply looks on the network, via Bonjour, for any Radmind servers, and when it finds them it gives you a list of available server IPs to chose from. Nice.

The next step is a bit more complex. It's time to build the MC. To begin, I am doing a fresh install of Tiger, running all current system updates (were on 10.4.1 as of this writing), and then setting this base install up the way I want it. A surprising amount of stuff gets set at this stage: Network, Energy Saver, Sharing, QuickTime Pro (license and settings), Accounts, and Security preferences all get set here. I am setting up my two local admin accounts at this stage as well. Also, I'm setting up binding to my Macserver for authentication, as well as installing my custom NFS mount Startup Item for mounting our home account RAID. Finally, I will install Radmind, of course. What I want to end up with is a very basic, clean system that represents the bare minimum installation for running in the lab, with no third party apps yet installed, and no customization of cron or any login scripts or anything, except Radmind, which should be set up as well, and should be part of the Base Install on the server so that it can create Radmind-controllable clones of itself, which can then be easily updated. This is my Base Install.

Once the Base Install is done, I will configure the machine to be a Radmind client that is controlled by the Radmind server. The MC will doesn't know it's the MC, and it doesn't really have to. In fact Radmind doesn't even need to be aware of this. The concept of the MC is really for us humans. So the MC will be set up as a client, just like any of the other clients. Essentially, Radmind will then begin using this machine to set up lists of files. These lists are what's really important. The lists will be used to compare files on various clients. Additional clients will be updated based on these lists.

At least that's how it's supposed to work.


Failure:I spent an entire day attempting to set up my MC with Radmind. You need to do two things on the MC before you can really get down to business with Radmind: 1) create a negative transcript for the server to use, and 2) create a positive transcript for the server to use. These are the most basic, fundamental lists that the server uses to compare against clients. The negative transcript is a list of files that should not be propagated to clients, and the positive transcript is a list of files that should get propagated. For some reason, I had endless problems creating these transcripts. The first problem was a discrepency between how the GUI application creates transcripts, and how it reads them, by default. The GUI app is set to "Begin transcript comparison from this path: / (slash)" whereas, the default transcripts created by the application use ./ (dot slash) at the beginning of their file paths. So, right out of the box, the Radmind GUI fails horribly, and my first, vanilla, Radmind-built negative transcript generated all manner of error message. Changing the defaults fixed it, and I was able to generate a useable negative transcript.

The second problem... Well, I don't know what caused the second problem. Basically, I can't seem to generate a positive transcript that will verify without errors. All I'm trying to do is create the base-loadset.T transcript, using all the defaults in Radmind, and each time I do, on my server I get a list of positive transcripts with numbers like "994" appended to the file names, and my base-loadset.T file generates an unspecified error when I try to verify it on my server. I've tried this numerous times, and the same thing happens each time. Frankly, I'm sick of it. Each base-loadset.T creation takes upwards of half an hour, as the client must compare and then copy the entire loadset (essentially, all the files on the hard drive) to the server. Multiple failures at this stage are infuriating. But what's worse is that there seems to be no way to modify the configuration once it's been uploaded. Making a new loadset with the same name gives me the error "Loadset exists." So the only way to re-attempt loadset creation, or modify a loadset, is to erase it and start over. For a system that's all about monitoring and tracking changes to systems, this seems like a backwards approach. In any case, after a day of trying, I still have not successfully created a working positive transcript.

There are lots of problems with the Radmind implementation. One irksome problem is the inconsistency of just about everything in the application. (I'm talking about the GUI here.) For instance, running through the setup steps frequently yielded different results, both on the server and on the client -- sometimes I'd get errors, go back, repeat and get no errors; sometimes, after setting up my negative transcript, I'd be asked to set up my positive, sometimes I wouldn't. Also, the interface is ridiculously inconsistent: while running the setup steps, pressing the "Go Back" button does not take you back to the beginning of the setup steps. WTF? Maybe they should rename that button "Go Somewhere You've Never Been Before," because that's where it takes you. Another issue I faced was in altering a transcript: the latest version of the Transcript Editor completely garbles your transcript if you add an item. I had to use a previous version to add items to the list. And there's more: Adding a server to your server list in the Radmind preferences does nothing apparently. Even after doing this I was always queried for my server IP. Also, once added, a server cannot be removed from the list. There is a "remove" button, but it does nothing.

This is why Systems Admins should never design software.

It's been suggested that I try using Radmind from the command-line. I am tempted. But the problem is that the Radmind CLI environment and implementation is so complex that I'm liable to spend a week learning it, only to find that it still doesn't work. I've already been online reading various Radmind mailing lists, and people are having all manner of difficulty there as well, particularly going to Tiger. I just don't think, at this stage, it would be wise to continue with this plan when the product is clearly so problematic on so many levels.


Resignation:So, after all this planning and testing, I've decided not to use Radmind after all. My reasoning is basically twofold: 1) Radmind is supposed to make my life easier, not harder. Thus far it's only introduced complications to my life and to the process of administrating my systems. And this is just in setting up my base system. What problems will I encounter when I start adding the many gigabytes of application files? Seems to me a product that is designed to simplify lab management should be fairly straightforward and easy to master. If it's not, what's the point? Using Radmind only adds an extra layer of complexity that I'm not even sure I really need. 2) Radmind is supposed to make updates and installs less error-prone and more consistent throughout the lab. But, again, the Radmind process itself is inherently error-prone, at least in my (and many others online) experience. How can I rely on such a system for lab maintenance with any confidence at all? I simply don't trust it. And if Radmind breaks with each upgrade of Mac OSX (which it might or might not, I just don't know, but indications are that it does), then again, what's the point? For all my work, what do I get?

There's got to be a better way.

Seems to me like the ultimate Radmind solution, at least in GUI-land (or maybe even as CLI solution -- why not?) would be something quite seamless to the admin. (And yes, I will now try to outline what I would like to see in a Radmind-like solution despite having said, not three paragraphs earlier, that Sys Admins should never design software.) I envision something like this: There is an interface called "Base Builder." Here you configure your base system, which would be your Master Client. On the MC you open Base Builder and tell it to use the "Current System" as your base install and it uploads everything to your server. I don't need to see a list of files at this point. Just build the damn system. Keep your lists to yourself, thank you. After the system is built, you can create your "Exclusions" from within the Base Builder app. This is real simple too. You just drop the folders you want to exclude into a GUI window, and then the properties of each exclusion. Now you've altered your base install, so you get a window that says something like, "Base install has changed. Would you like to rebuild the base install on the server?" and a big, fat "Yes" button. Hit "Yes" and your changes are propagated to the server. Simple. And when it's time to add applications or other layers to your base, you go to the "Layer Builder" interface. This is similarly easy to use. Here, you tell the app where to find your base install, or you can say "User Current System." Layer Builder will take a snapshot of the base install and keep that snapshot. You'll install your apps, then tell Layer Builder to create the new layer from a comparison between the base install snapshot and the new system. Layer Builder will allow you to name your new layer, and then save a snapshot of the layer. Finally, you'll have a "Sets Builder" interface. Here you can combine various sets of layers to create different configuration for different machines. This would have three panes: A "Layers" pane, a "Sets" pane and a "Computers" pane. You'd drag layers from the Layers pane into sets in the Sets pane to create your various configs. Then you'd drag sets to computer lists in the Computers pane. In the Computer pane you could run "Compare" to see differences between the actual computer's files and the files in the set. And if there were differences, you could propagate them to the client using something like, oh, I don't know, an "Update" button, let's say. And that's it, basically. For advanced users, you could look at the file lists and make changes between the server and the MC and the various clients. This seems like a key missing feature in Radmind. The ability to change the base config, or any of its transcripts in any meaningful way seems to be absent. These sorts of changes require rebuilding everything, which takes a great deal of time and effort, and is error prone. And isn't the whole point of this to make the process of lab maintenance easier and less error prone?

I realize that what I've described is essentially what Radmind is and does. But Radmind does it in such an abstract and confusing way that the process becomes needlessly complex and defeats its own purpose. Something that companies like Apple understand -- and this is a large part of why I prefer the Mac platform -- is that good visual design and clear language can make an interface, or even a CLI app, a breeze to use, and that that is actually the point of GUI applpications: To make a difficult and confusing process clear and intelligible. Radmind's visual interface is a mess, and its language is dizzingly obscure. Here's a list of terminology, for example: negative transcript, positive transcript, command file, loadset, base loadset, overload, configuration. Here is a list of some of the files involved: negative.T, base-loadset.T, base.K. The files that end in .T are transcript files, which are lists of files belonging to a loadset. Get it? Of course not. Who would?

It's totally ridiculous.

Now that I've gotten that off my chest, I need to come up with a good way to proceed with my lab update. And although I am scrapping Radmind for now, I would still like to think of a way to ease future updates and remove the inconsistencies from the way I've done things in the past. There are a few options here. These involve disk images and databases for the most part. In any case, I will be giving this some serious thought as I move forward. But these issues will be the topic of a future article.

Labels: Lab, Systems, Tiger, TigerLabMigration

As foretold, I've rebuilt my admin machine with fresh-from-the-factory Tiger. Nice. Things are going well.

I'm testing right now. Partly, I'm testing how Tiger does from a clean install. I'm also testing its interaction with my Panther Server. So far I have hit one, minor snag here, and it appears to be a bug in Tiger's Directory Access application.

In Panther Client's version of DA, the LDAPv3 configuration panel was set up by double-clicking the "LDAPv3" entry under "Services," then clicking "New..." at the bottom of the drop-down panel, and entering your server info in the available fields of the new entry. The Tiger version is slightly different: In Tiger, pressing "New..." brings up a dialogue box called "New LDAP Connection." Here you can enter information about how you want your client to use LDAP (i.e. for contact, or authentication, and whether or not to use SSL encryption). At the bottom of this dialogue is a "Manual" button which allows you to set up the panel the old, Panther-style way. Being old-school, I chose "Manual."

Silly me.

Turns out, setting up the server binding with the "Manual" button works, but the settings don't survive a restart. I tried this numerous times, and it would initially work, immediately binding to the server (which, by the way, is still Panther, and this may be part of the problem, but I seriously doubt it). But after restarting, though the entry would still exist in DA, the binding would be broken. No authentication, no computer management, nothing. The way to get it to work is to use the new, and supposedly improved, dialogue that pops up when you press "New..." in the LDAPv3 configuration panel -- the aforementioned "New LDAP Connection" window. Using this method to set up binding to my Mac Server worked. The new entry, once created in this manner, can be edited after the fact if need be. And best of all, the binding survives a reboot of the client.

Actually, the new dialogue does have one really nice thing going for it: It sets up the server paths in "Authentication" and, if you tell it to, in "Contacts." You used to have to set these up manually, in a separate steps, under the "Authentication" and "Contacts" panes, but now, if you check the appropriate checkboxes, DA does it all for you from the "New LDAP Connection" panel. Nice.

Well, it would be nice, if the "Manual" method worked. Still, it's better than a kick in the teeth.

So, with authentication working between Tiger client and Panther Server, I'm over halfway there. In addition to authentication, preference management (things like Login Items and such) appear to be working. The last thing on the client/server relationship checklist is services, mainly printing services. I'll be kicking this around for a bit, and then I'll be starting my Radmind work.

Just a preview of that: To start, I will install Radmind on my admin box and set it up a a Radmind server. Once that's all up and running (and I should really do some checking up to make sure Radmind is Tiger compatible), the odious task of setting up the master client from scratch will begin. To reiterate, this will be a machine that has everything that I plan to install on the various workstations on the floor. From this, sets will be made for each hardware/software configuration. Each change to the master client will have to be tracked by the server, starting with the base install, and building loadsets (overloads?) as software is added. This will take some time and patience, but it should be worth it in the end.

Let's hope.

So, when next we meet, I will be setting up Radmind. I will do my best to be as detailed and comprehensive in the documentation of this process as I possibly can be.

Oh yeah, and one other thing before I install Radmnd: I'm cloning my working Tiger install. 'Cause you never know.


UPDATE 1:
I have begun building my Master Client machine. The problem of server/client binding failing after reboot is now occuring on this machine, and using the "New LDAP Connection" window to set it up does not seem to work in this case. Thus far, I have been unable to bind my Master Client to the Macserver in a way that the binding survives a restart. This would seem to be the salient error in the system log:
Jun 27 19:43:45 systemsBoyMac /System/Library/CoreServices/mcxd.app/Contents/MacOS/mcxd: DSOpenNode(): dsOpenDirNode("/LDAPv3/192.168.1.10") == -14002

Simply opening the Directory Access application, authenticating, and opening the LDAPv3 configuration panel will bind the client to the server. The message that gets written to the log in this case is:
Jun 27 19:50:02 systemsBoyMac /System/Library/CoreServices/mcxd.app/Contents/Resources/MCXCacher: CacheUser(0, systemsboy) == -14136

I don't get it. I'll post when I find the solution.


UPDATE 2:
Deleted the /Library/Preferences/Directory Access directory -- which contains the preferences for, obviously, Directory Access -- rebooted (twice) and was able to login as a network user both times. I'd done this before. The differnce this time was that I logged in as a network user first, before logging in as a local user. I'm making a third attempt this very moment. The computer is rebooting... Now logging in as a local user... Good... Now as a network user... No luck! The login screen shakes it head at me. It would appear that setting up binding, rebooting, then logging in as a local user will break the binding. Man. That's fucked up.

More to come...


UPDATE 3:
So now I've trashed the DA prefs again. But when I launch the DA application, it's still set up with the old binding config. So apparently, there's some pretty nasty caching going on. I'm trashing all prefs in /Library now, and clearing the mcx_cache from NetInfo. Rebooting. Setting up the Network. Setting up DA. Rebooting. Now I'm bound, and network logins work. What a blast. I hate this shit. Trying again -- rebooting, logging in as a network user, success. One more time, this time local user logs in first -- reboot, wait... Binding is broken... WTF!? I must admit, I'm stumped.

Will keep you posted...


UPDATE 4:
Okay, now this is too bizarre. There are telltale signs that the client is not bound. One of them is that the "Other..." button does not appear in the list of users at the login window; there are only local users in the list. The other is that the server does not appear in /Network/Servers. When the client is bound, you will see a network mount (or actually, a symlink to the network mount) in this directory. So I'm poking around in the Terminal on my client, with the /Network/Servers window open, and I'm looking at logs and whatnot, and all of a sudden I see the server mount point show up. Out of nowhere. And it occurs to me, maybe the binding is just slow. So I reboot the client and just leave it at the login screen. After about three minutes, the "Other..." button shows up. Just pops right up. Client is bound.

I can reproduce this on two machines now, but I can't explain why it takes so long, and why this behavior is so inconsistent. My server is at 10.3.8 and has some strangeness about it. I will be looking at it to see if these problems are perhaps now manifesting themselves more obviously with Tiger clients. I will also consider upgrading the server to 10.3.9, as there are apparently changes to the LDAP schema that were made with that update that were in preparation for Tiger.

I'll let you know what I find...


UPDATE 5:
Another anomaly: Logging in to a bound client and navigating to /Network/Servers and selecting the server automount (we have a home account automount) causes the Finder to beachball indefinitely. Relaunching the Finder kills the beachball, but the network mount is broken (i.e. there is an empty mount point), and lots of automount errors in the system log. Oy! Time to fix/update the server.

I am now cloning my Macserver in preparation for moving to 10.3.9. Hopefully this will at least resolve the automount issues, and maybe even the slow binding issues. Of course, the ultimate solution will be to migrate to Tiger server. Still no idea when I'll be getting my disks, but when I do, there's a great article on the migration process at AFP548.


UPDATE 6:
I have updated my Panther Server to 10.3.9. Though the upgrade went smoothly, and I have experienced no problems with it, it has not solved my Tiger client problems, which are, to reiterate:
1. Binding to the server from a Tiger client takes approximately three minutes after reboot to occur. So there is a three minute period after a reboot during which a network user on a Tiger client cannot login. Suddenly, after three minutes or so, he/she can.
2. Network home account mounts from the Panther server do not automatically mount on the Tiger client. They should, and they do in Panther client. Here's the error message from the Tiger client system log:
Jun 30 16:42:18 systemsBoyMac automount[241]: Can't mount pantherServer:/Volumes/FlashDeveloper on /private/Network/Servers/pantherServer/Volumes/FlashDeveloper: Authentication error (80)

It's looking more and more like I'm going to have to install Tiger server before I can proceed much further.

Shit...


UPDATE 7:
Well, this is turning out to be more fun than a barrel of monkeys.

Finally got the network home account mounting and the user authenticating. It is not an automount problem, but rather an authentication problem. Apparenly, Tiger client cannot login to Panther server if the user's password is of type "Open Directory." Crypt passwords work fine. (I knew there was a reason I wanted to stay with crypt, but nooo, Apple said "Use Open Directory passwords. They're better." Yeah right.

I'm on my way to the Apple Discussions to see what I can dig up.

I'll let you know...


UPDATE 8:
(Hey, that rhymes.)

So I changed the password type of my network user to "Crypt" and could suddenly log in, right? Now here's something weird: I changed that same user's password back to "Open Directory," and guess what? It worked.

This would seem to indicate something screwy with the password server, but I'm hard pressed to say what it is. In any case, this presents a real problem for me, as I have about 50+ users with Open Directory passwords, and the way things are right now, they're not going to work in Tiger. The only way for me to change them is to get all 50 users to come in and change their passwords in Workgroup Manager, and that's just unacceptable to me.

This is why I long for a utility that lets OD admins change user password types without having to reset the password. This doesn't seem like a stretch to me, nor does it seem unreasonable, particularly if you have a lot of users, which I do. Because now I'm stuck with 50 or so users with Open Directory passwords that just won't work, and the only way to fix this is to reset their passwords, when really, all I want to do is change the password type to "Crypt," and all the data I would need to do that (i.e. the passwords) is there on the server. It's just inaccessible by the admin. So fine, give me a utility that lets me change the password type without resetting the password. I don't need to see the password to do this; the utility can do it. I just want to make the change, and I can't, and that's Bad. (And, BTW, the reverse could be true as well: What if you have 300 crypt-style users and you want to change them to Open Directory passwords? As it stands, I guess you're going to have a mightly long line outside your door.)

Anyway, from what I've read at the Apple Discussions, this sounds like it might be a problem not just with Panther servers, but with Tiger servers as well. I'm betting these are all upgraded or migrated servers, and that fresh Tiger (and maybe even Panther) server installs work just dandy, which is why only some people are experiencing problems.

Anyway, this is endlessly annoying. I'm done for the night...

Labels: Lab, Systems, Tiger, TigerLabMigration

Snags! I've hit snags! And in the most basic part of this migration.

I don't know what it is, but it sure seems like any time I want to do something ambitious I end up having the weirdest problems. This lab migration is no exception. My first step is cloning my system drive, in case I need to revert back to the previous working state for some reason. A tedious but necessary precaution. And drop-dead simple, right? I mean, how many drives have I cloned in my lifetime? Well, I don't know for sure, but I lost count somewhere around twelve-bajillion or so. And how many times have I had a problem with it? Maybe three. And usually it was because I was doing something stupid.

But today, of all days, when I'm finally ready to take the plunge, wipe my system and install the dreaded Tiger, I find myself unable to successfully clone my drive to a disk image. Everytime (and I've tried Carbon Copy Cloner and Apple's Disk Utility) I get the same error message telling me that the disk image/folder is too big. Too big for what? I'm cloning a 13 GB volume to a 230 GB hard drive. Seem like I should have plenty of space. The error occurs, in all cases, right after the initial sparse image is created, and right before the ASR scan/conversion begins.

I tell you, I'm stymied.

I'm on my fourth attempt at this point, and this time, rather then being booted from the root drive, I'm mounting the system in firewire target disk mode, and cloning on a seperate system. (See? The advantages of having a lab full of computers. Nice.) My reasoning here is that I'm worried that there's something terribly wrong with my boot drive, and it's confusing the hell out of hdiutil. (Did I mention, the error message is from hdiutil?) So I figured I'd try from a presumably happy, healthy boot drive and see what happens.

I've been doing this all day, and it's getting pretty old. So, while my attempts at cloning run I am also:
1) Writing this blog (obviously)
2) Reading other blogs
3) Installing Red Hat Linux on a Windows box.

I am surrounded by progress bars, and yet I can't seem to make any progress.

Systems work can sure be frustrating sometimes.

Oh well.
_______
Update 1:
Argh! It happened again! Below is a screen capture of the error message.

Also, SprintPCS has been down for two days doing "maintenance." Probably removing cool features and replacing them with shitty ones. This seems to be a general tren in the industry. But I'll tell you, if I took so long to do maintenance, I'd be fired. What this means for me, of course, is that I cannot upload any photos.

Apparently, I can't do anything today. Maybe I should just go home.

Again I say, Argh!

_______
Update 2
Well, I've figured out the problem. Should've just googled that error message in the first place, but no, that would've been far too quick and easy. The problem is a bug in hdiutil and OSX 10.3.9 that prevents creating an asr restore image of disk images over 8 GB. So right this very moment I'm making a lean OSX 10.3.2 boot disk on a 6 GB firewire drive I have lying around. I'll boot off that drive to make my restore image. Hopefully, that will do the trick and I can move on.

Hmph! I knew there was a reason I didin't want to upgrade to 10.3.9. And there it is.


Um... WTF?

Labels: Lab, Systems, Tiger, TigerLabMigration

First, a very little about me: I run a Mac lab at an art school. I am in charge of about 30 Macs that are used for a huge variety of things, including office work, graphics, video, interactive authoring, web authoring, the teaching of these topics, and just about anything else you can think of. I manage multiple hardware configurations in an extremely heterogeneous network that consists of Windows and Linux machines in addition to my Macs. I run two primary servers: a Quicktime Streaming Server, and a Macserver that handles login authentication and other network services (including print services and preference management and maybe a few other things I'm forgetting offhand). I also run a test server or two, and soon I'll be running (if all goes well) an LDAP replica for my Macserver.

Phew! Okay...

As the administrator of this lab, I am constantly being asked if I will be upgrading to the latest, greatest Mac OS. This Summer is no different. And I just want to outline, if only for myself, some of the pros and cons, as well as how I might proceed, in doing so this year.

First let me say, the release date of Tiger affords me the perfect opportunity to upgrade. Previous revisions of the Mac OS always seemed to come at odd times, often in the middle of the semester, and made it inconvinent if not downright stupid to upgrade immediately. Usually what I would do in these instances was test the new OS and then, depending on how important the upgrade was and how smoothly the transition could be made, either upgrade between semesters or wait until Summer. Summer, you see, gives me a full three months to plan, test, and implement an OS upgrade. So I'll say right here and now that the release date of Tiger makes it almost certain that I will upgrade this Summer. Tiger is complex enough that it's not something I'd like to attempt between semsesters, when my time is much more limited. Waiting would likely postpone an upgrade to next year, and that's unacceptable to me, and would probably become unacceptable to students and faculty in the near future, when the real Tiger benefits start making themselves more apparent.

What I mean by this is that, from everything I've read, a lot of the really good, juicy, exciting changes to Tiger -- the things that will really be a boon to users -- are low-level. For instance, the offloading of many tasks to the graphics card. While these changes might not be readily obvious, or even useful, at the moment, when new apps like Final Cut and Motion begin shipping, I think we'll see some really good reasons to upgrade, and I'll be kicking myself if I haven't. Worse, my students will be kicking me. (Yowch! That's a lot of kicking.) I suspect Tiger will be really good for video. And we do a lot of video on our Macs. With the FCP Suite soon to hit the streets, I've one more big reason to upgrade.

The low-level changes, however, are also what make this, in my mind, such a challenging update. I've installed Tiger on my Powerbook for testing using the "Archive and Install" feature, and I must say, I've been underwhelmed with the performance of the new system. Most people are claiming that Tiger brings with it performance gains, but I've not seen them. If anything, my Powerbook seems a bit slower than it was on Panther. Mind you, I haven't done a clean install on that machine since I bought it (three years ago), so there's probably plenty of old junk that needs cleaning out (like, maybe, all those old, non-binary preference files) and I can't help wondering if an "Erase and Install" would have been the better route. Nevertheless, I may prove far too lazy to ever attempt such a process, as re-installing all my apps would take days, and I just don't have days.

Enter The Lab.

The great thing about working in education (or one of them anyway) is that I get to test and try stuff out on someone else's hardware, and I have a lot of machines at my disposal upon which to do so. The rationale for all this is that I will then implement, based on these tests, and build a productive and efficient lab for all to use. Ideally, the whole lab benefits from my test experiences. So, though I may not be inclined to Erase and Install on my Powerbook, or test generally on my personal computers, The Lab is the perfect place to try such things.

The thing about Tiger is that it's complex. It's a beast. There's a lot of stuff that I've implemented in Panther that will break instantly, and a great deal more that will need serious modification and coordination. I use rsyncx for backups of staff machines, for instance. What is the best way to seamlessly preserve that functionality given the facts that A) Tiger introduces a whole new, resource-fork-aware, version of rsync, B) the staff machines will probably be among the last to get upgraded, C) my machine, which performs the backups, will be the first to receive the upgrade, and D) the versions of rsync must match between client and server? It's a chicken-or-egg problem that will require a hack to workaround, but it's do-able. But what about my Macserver? What will be the interaction there? Should I upgrade the server first and then the clients, or vice-versa? (I still have not received my copy of Tiger Server, BTW, so that pretty much answers that question.) There are a whole host of questions and problems like these that will require some serious testing and planning.

So, at this point, I've pretty much decided on a first stage of my migration plan. And since Tiger is so complex, I've decided to shake off my complacency and implement this upgrade boldly, and I might add, from scratch. Yes, scratch.

Here's The Plan.

There are a few things I've been wanting to do that this Tiger-migration-from-scratch really gives me a good chance to do. The first thing is repartitioning. In the past I've partitioned my lab computer drives into two partitions: a SysApps partition that holds, what else, the system and application components, and a Work partition from which students can, yes, work. With the ever-increasing list of applications installed on my systems, and their ever-increating size, my SysApps partition is getting a little cramped at 20 gigs. And with hard drives getting bigger all the time, and students primarily relying on their firewire drives or network storage at this point, it seems like a good time to repartition. I'm thinking that SysApps will grow to 50-75 GB in the new scheme.

The second thing I want to try is to implement Radmind to track and monitor system configurations and to apply updates over the network. This may be overkill given the size of my lab. Radmind is really meant for managers with hundreds of computers and far more resources than I have at my disposal. And it's really not necessary for a lab of thirty or so Macs. Still, I see an opportunity here to try something new and learn. But also, there may be a real advantage to implemeting Radmind, for a lot of the reasons I complained about with regards to this and past upgrades. Seems to me like Radmind could effectively take some of the sting out of upgrades. Radmind allows the admin the ability to create sets of updates to both applications and the OS in what amount to layers on top of the base install. Ideally I want to create a system whereby updating the entire lab to a new OS revision is as easy as a few mouse clicks, so that after I've gone and tested Leopard, upgrading the lab will simply mean upgrading one machine and then porting those changes to the Macs in the lab. And, should there be a major problem I've overlooked, reverting will be just as fast and easy. Again, this is best handled from scratch, starting with a clean install of the OS for the base config.

Finally, if my Powerbook is any indication, I think that Tiger will be a much smoother upgrade going from scratch, and doing this gives me an opportunity to rethink, to some extent, my current implementation of the Mac lab. Things are working pretty well, but they could always be better.

Where to begin?

The lab admin is always Guinea Pig Number One, so I'll be starting, of course, with my machine. Yup, that's right. Today is the day. I'm wiping my admin box and installing Tiger. Needless to say, I will be backing up my Panther system with Mike Bombich's wonderful Carbon Copy Cloner, just in case I need to get back to a fully working system. If I could, I'd clone a working copy to a spare firewire drive and boot off that if I needed to, but alas, the downside to the smaller educational environment is that you don't have endless hardware resources, and so, no such drive is available to me now. I'm going to have to really take the plunge here, making a clone image for backup, and if necessary, restoring over my fresh Tiger install. This will be good motivation, however, to stay with Tiger unless it's absolutely imperative I revert.

After wiping and installing Tiger, I'll need to get backups online as fast as possible. This is my first priority. I will need to continue to use rsyncx as a stop-gap until I can get staff machines upgraded. Upgrading staff machines will be the most problematic and scary, as these machines are in continual use throughout the Summer, and as they have data and applications that absolutely must be preserved and in working order as fast and as seamlessly as possible (yet another reason to make sure my backups are working before proceeding). That being the case, I may opt to use the "Archive and Install method on those systems, but that decision can wait a bit. One potential major fly in the ointment is the possibility that rsyncx won't work in Tiger. If that's the case, I will need to upgrade my staff machines sooner (read: much sooner) than later. Like fast!

Next, I should probably start installing applications, just to get back up to speed. But this introduces an interesting problem: Perhaps my next step should be to install Radmind, test it, and then use my machine as the base config. My machine is different than all the other lab machines, though, so actually it might be unwise to use it as a model upon which to base all the other lab Macs. Using an admin machine for this model also seems like a bit of a security risk, so I probably won't begin with Radmind on my system. I think the best way to proceed will be to get my apps installed and get my system to a useful state again, then start building my Radmind server, modeling the base config on a lab workstation, which will also need to be built from scratch. This will require a lot of extra work, but I believe it's the best way to go. In any case, this is a really good example of the kind of logistical problems involved in a major upgrade such as this.

Once all my important apps are installed and my machine is doing the things I need it to do for work, then I can begin hammering Tiger for faults and problems some more. At this point I'm pretty familiar with all the changes, so this stage shouldn't take too long. The trickiest part will be getting things like cron set up again and porting over all my Startup Items from my backup. (I have some custom Startup Items that do lab-specific things that I'd really rather not delve into here. Suffice to say, these things, too, will likely require some testing and tweaking.)

When everything on my system is back up to speed and working to my general satisfaction, it will be time to test how Tiger client inteacts with Panther Server. That is, can I still log in as a network user? Do print services still work? What about managed preferences? If the answer to any of these questions is "no," then I need to figure out the best way to migrate my Macserver to Tiger before updating the lab's workstations (at which point you can expect another article). If the Tiger workstations can be peacefully managed from the Panther Macserver, I can wait a bit to upgrade the server. This would be optimal, and there is a good chance that this will be the case. I've already tested logins, and I know they work. Hopefully the rest works too, with minimal changes, as I'd like to (and may be forced to, as I have no idea when to expect my Tiger Server CD) update the server last.

And just an aside on that last comment: The logic behind upgrading the server last, I realize, is a bit skewed. The main idea is that, if I upgrade the server first, it is more likely to break my clients than if I upgrade the clients first. Though this may not actually be the case, my primary concern here is user logins. If those break, I essentially have no lab. Other services I can live without, but user logins are paramount. And I already know they work, so it's a much safer bet to upgrade the clients first. Also, there is the plain fact that I have no Tiger Server CD and I want to get moving. Finally, doing the client side the way I am is a much bigger job than the server is likely to be, and I want to get started on the migration now. In a perfect world, I suppose I'd build a Tiger Server on a seperate machine, and it's client on yet another machine and do my test builds in an all-Tiger environment. Alas, that is just not possible for me at this time. I am forced to be a bit more messy than that. But this is the only way I can move forward right now. And it should be fine.

Okay, so Tiger is installed on my admin box, backups are going again, apps are up and running, admin stuff is generally working. Great. Now it's time to start building my Radmind server. (And let me just say that, before I proceed, I may clone this fresh, working version of Tiger to a disk image, just in case.) The Radmind setup process is twofold. For the first part, I need to install the entire Radmind package on my admin machine and set it up as a server. (I will not go into Radmind setup in this article, as I don't know it nearly well enough to describe it from memory, but perhaps in another article.) For the second part, I need to begin building my master client. Ideally, the master client will be a conglomerate of all the various configurations in the lab, built in stages. Each stage represents an installation set. This will be one heavy machine. Essentailly we'll install everything we have on it. It will also take any new updates in the future. These updates will be tracked as well. Finally, sets of installs for various configurations of Macs can be built. For instance, some of our Macs run Motion. Motion will get installed on the master machine, and any machine that runs Motion will be put into the "Motion Computers" list, which defines what computers get what software/updates. To install Motion, or anytime it's updated, we install on or update the Master client and tell Radmind to update all the computers in "Motion Computers." Cool! So this Master client needs to get built first, and very carefully, tracking each change along the way with Radmind. It's probably a good idea to start with a list of the various system configurations in the lab, i.e. which systems get which software. This will eventually become the "Computers" list(s) in Radmind.

(By the way, I you might be wondering why I don't manage all this from Tiger Server's "Software Update Server." Well, from my understanding, Tiger Server only updates Apple software on clients. Radmind can update anything. Plus, Tiger Server doesn't let you revert changes, which is one sweet feature of Radmind. Radmind is very complex, but vastly more powerful in what it does, as it is specialized for that application. Tiger Server is really a different beast that will probably never be able to do everything Radmind does. If it ever can, it will probably be because Apple has bundled Radmind with Tiger Server. Which I could see happening someday, actually.)

Okay, this is where things get a bit hazy. But that's alright, because this stage won't be happening for at least another few weeks, and by then I'll have a lot more of the information gaps filled in my roadmap. The final stage will be to wipe all the machines on the floor, repartition them, install Tiger on them, and then install Radmind on them. Actually, I'll probably just build one general system, set it up as a Radmind client, and clone it to my remaining systems. Once all that is done, it's time to whip out my Radmind. At this point it should just be a matter of designing sets in Radmind for each hardware/software configuration in the lab, and then simply telling Radmind to setup the systems. The staff machines can get updated then too, or along the way (though I may or may not manage them with Radmind -- staff machines are still up in the air at this point, and if I do use Radmind on them, I may make a completely new Staff Master config). And then I can turn my attention to the server. But that shouldn't be too bad, right? (I can't believe I just said that!) The major hurdle is the lab migration and Radmind implementation. Once that's under way, life should be pretty good.

Sound easy? My life should be so easy.

I'll keep you posted.

Labels: Lab, Systems, Tiger, TigerLabMigration