SysAdmin1138 Expounds

A disturbance in the force.

2009-11-19T18:07:00.002-08:00

A friend of mine's experiences with GIMP vs Photoshop are telling. Like many, she tried switching but found GIMP less than useful for any number of things. Such as no 'draw a hollow square' tool, among many, many others. When poking the developers about this the reply came back, in essence, "GIMP is not a Photoshop replacement, it's for photo manipulation."

Well, it seems that the Ubuntu distribution managers agree with my friend more than the GIMP developers, as they're dropping GIMP from the default install. Why? Well...

the general user doesn't use it
its user-interface is too complex
it's an application for professionals
desktop users just want to edit photos and they can do that in F-Spot
it's a photoshop replacement and photoshop isn't included by default in Windows...
it takes up room on the disc

If the most popular desktop linux on the planet calls GIMP a Photoshop replacement, then... it just might be a Photoshop replacement. No matter what the Devs think. It will be interesting to see what openSUSE and Fedora do in their next dev-cycles. If they keep GIMP, things will probably continue as usual. The same if a user revolt forces it back into the Ubuntu defaults. On the other hand, if Fedora and openSUSE follow suit, this will be a radical change in the GIMP community environment. They may start addressing the UI issues. Who knows.

As it stands, Adobe has nothing to fear from GIMP. Anyone well versed in Photoshop will find the UI conventions of GIMP wildly different, and the same applies to methods to solve certain image problems(*). Adobe only needs to fear the people who a) don't want to pay the umpty hundred bucks for Photoshop, b) aren't willing to pirate a copy, and c) are willing to tough it out and learn a completely new package with radically different UI metaphors. There aren't that many of those.

Me? I've never used Photoshop. Or if I have, it was back in the early 90's. I've been using GIMP all this time and that's what I know. I intend to keep it that way since I really don't want to start paying Adobe all that money. That said, I totally understand why people don't like it. I also miss simple tools like 'draw a square', and 'draw a hollow circle'.

(*) Some of these solution paths are patented by Adobe, so no one else can do it that way if they wanted to. This is what closed source software brings you.

Restrictive internet policies

2009-11-17T12:46:00.001-08:00

A friend of mine griped today:

In a stroke of utter WTF-ness... my workplace has blocked access to LinkedIn.com.

It's not so WTF for me as I can see why it was blocked. LinkedIn is seen as a tool for people looking to transition jobs. So if you're blocking Monster and Dice, then LinkedIn is right up there with it. The fact that it also is a useful way to network for business is beside the point. From earlier gripes, this particular workplace is on a crusade to block all social-networking sites. I only saw this post because of email-to-post gateways, and they haven't blocked gmail yet.

It is situations like these that give rise to the scenario I described back in June: I Want my SSH. Additionally, a lot of social networking sites are publishing apps for the various app-driven smartphones out there. For users willing to invest a bit of money into it, corporate firewalls are no longer the barrier to slacking they once were.

Packet size and latency

2009-11-16T09:58:00.002-08:00

The event-log parser I'm working on has run into a serious performance wall. Parsing 60 minutes worth of security events takes 90 minutes. The bulk of that time is consumed in the 'get the event logs' part of the process, the 'parse the event logs' portion takes 5% of that time. Looking at packets, I see why.

I'm using Powershell2 for this script, as it has the very lovely Get-WinEvents commandlet. It is lovely because I can give it filter parameters, so it'll only give me the events I'm interested in and not all the rest. In practice, this reduces the number of events I'm parsing by 40%. Also nice is that it returns static list of events, not a pointer list to the ring-buffer that is the ususal Windows Event Log, so $Event[12345].TimeCreated() will stay static.

The reason the performance is so bad is that each event-log is individually delivered via RPC calls. Looking at packets, I see that the average packet size is around 200bytes. Happily, the interval between RPC-Response and the next RPC-Request are fractions of a millisecond, and the RPC-Response times are about a half millisecond so at least the network is crazy-responsive. But that packet size and the serial nature of this request means that the overall throughput is really bad.

If there were a way to phrase the Get-WinEvents command to only populate the attributes I'm interested in and not any of the others (such as .Message(), which is the free-text message portion of the event, quite large, and noticibly laggy in retrieving), it could go a LOT faster. Since I don't have powershell installed on my domain controllers right now, I can't see how much running it directly on them would improve things. I suspect it would improve it by a lot since it should be able to use direct-API methods to extract event-log data rather than RPC-based methods.

As it is, we may have to resort to log-shipping. With over a GB of security event-log generated per DC, per day, that log server is going to have to have large logs. It has to be at least Vista or Windows 2008 in order to get shipped logs in the first place. Sad for us, we don't really have any spare hardware or VM space for a new server like that.

And finally, yes. There are 3rd party tools that do a lot of this. We need something both free and scalable, and that's a tricky combination. Windows generates a LOT of login/logout data in a network our size, keeping up is a challenge.

Passwords

2009-11-12T09:08:00.002-08:00

Over the years I've heard variations on this complaint:

"I don't need a secure password since everything I work on can be seen with a freedom-of-information-act filing anyway."

In the run up to the internal lobbying effort that allowed us to start password aging and put password complexity rules into place, we ran L0phtcrack against our Windows domain passwords. The results were astounding. A crushingly large percentage of passwords were still set to ones well known to be used by the helpdesk during password resets, users had never gone back and changed their password after having it reset by said helpdesk. A not much surprising but still disheartening number was the percentage of passwords set to either "password" or "p@$$w04D". These results are what convinced upper management to push password complexity policies onto the unwilling masses.

But that doesn't address the complaint above, merely shows the effects of this attitude. While it may be true that you work on nothing confidential, you still have one thing near and dear to your heart that you do care about; Identity. Especially with the advent of web-based Enterprise email, this is a very important thing. While it is trivial to impersonate an email address, it carries far more weight when that email is delivered from our servers. What's more, the ability to reply to legitimate emailas you is something you don't want. And finally, I don't know a single person that fails to have at least some personal correspondance in their work mailboxes, even if it only exists in the trash folder. That information may still be retrievable by an FOIA filing, but the generation of information does not, and generation of information is what you allow by having your password compromised.

We mean that. We don't allow managers to have departed employee's passwords for the same reason. Happily these sorts of gripes are becoming ever less common as the lessons of Phishing come home to more and more people. But this gripe is one that is particular to the public sector, so many of you may not have heard it before.

The Firefox anniversary

2009-11-09T13:58:00.002-08:00

Firefox turns 5 today. I'm sure you already knew that, what with it being widely covered industry-wide and all. This has caused me to look back on my own usage of Firefox over the years.

In the beginning, there was Mozilla. And I used it. And it was good. It had a nifty integrated html editor that I used on occasion. And I had used it for many a year.

I noticed the dev-work on Phoenix/Firebird and used it a bit at home on my Linux machine. Never did any serious browsing with it, but I did use it.

And then there was Firefox. When Mozilla announced that they were killing Mozilla-the-browser and replacing it with Firefox, I dutifully switched to that for day to day usage. I believe that was the 1.0.

And then there were the fights. Firefox did things differently than Mozilla did. I tried to take things in stride, but it was hard. Cookie handling was a big pet-peeve of mine (since remedied). The other one is still true.

I flirted with Opera briefly, but it was annoying in different ways. Sad.

And then there was the breakup. Which I blogged about here. You see, I'd learned about SeaMonkey, which is an OpenSource project aimed at bringing the defunct Mozilla-browser into the future. It had the experience I was used to, and worked with most of the Firefox extensions too! What's NOT to like? I was hooked and made the switch. Good by, Firefox! Won't miss you.

And then I moved to openSUSE as my primary desktop. This required a certain amount of Firefox usage simply because that was the 'built in' browser. Mostly I ignored it, since they had SeaMonkey as an option.

And then SeaMonkey started getting stale. The same UI for, like, 5 years gets old. And the little bits where it differed from the IE/FF experience were growing. So I started using FireFox on the side at work, as a way to do things like run my Google apps in a separate browser so I could do all of my other searching without directly associating my search terms with my Google account.

And then Firefox 3.5 came out. And it sucked less. I converted to FF3.5 at home, but still kept with SeaMonkey at work. It still involved some nose-holding in various spots, but I was determined to bull through. I got used to the popularity contest in the drop-down bar. I still miss the way typed in (or pasted in) URLs never show up in that list, but I got used to it.

And then SeaMonkey looked to be in PermaBeta for 2.0. Knowing I am a very small minority of web users by using SeaMonkey (0.58% of viewers of this blog, which is less than the 2.08% of you still using Mozilla), I had doubts about the long term prospects of SM. My Firefox usage ticked up again. And when Opera 10 came out, I gave it a real going over. For work stuff it didn't cut it, but it just might for home use.

And then SeaMonkey 2.0 actually released. Download it now! It integrated some of the more annoying-but-need-to-have features of Firefox (like the SSL handling) but kept the drop-down sort the way I like it. An MRU list.

And that brings me to today. At work Firefox is the browser I keep logged in to Google for various things, and still use SM for all of my other browsing. I find that handy.

And now you know.

Audit logging

2009-11-05T15:10:00.001-08:00

When I first arrived here we used to get this question four or five times a year:

Can you check to see who was logged in to server X at 2:34pm yesterday?

Back in 2003, "Server X" was 98% likely to be a NetWare server. In 2003, Novell hadn't come out with Nsure Audit yet, so the only such logging available were the NW4.11-era text-mode audit logging. Which, to put it politely, didn't even come close to scaling to our levels of access. Logs like that take a lot of space. A LOT of it.

Fast forward a few years, and we're now doing a heck of a lot more Microsoft networking. The domain controllers have security auditing turned on by default. While a day's worth of logs are smaller than the Novell logs would have been (not sure about NSure Audit log sizes, never got a chance to use them), it's still very large. A gig a day is not unreasonable, if not more.

One thing that MS auditing doesn't give us is 'lockout address'. So when a student walks up to the helpdesk and asks, "why am I locked out?" the helpdesk staff and look and see what IP did the locking. We can't do that right now on the Microsoft side. I'm attempting to fix this, which requires creating a log-parser for windows.

Happily, this is doable with PowerShell. Unhappily, it means 1.8 million events to chug through when I parse said log. Even more unhappily, the key data I want (IP, Username) is not in a straight up field and requires parsing the Message text. Any time you parse text like that, you become vulnerable to text format changes. It's not the ideal solution, but its what we have.

Once this is done we'll even have a lockout history which we didn't have before. So we'll be able to tell patterns like having a lockout 7 minutes after turning on their Mac, repeatedly. But first I have to finish writing it.

Novell federates with Google

2009-11-04T12:29:00.002-08:00

It seems that Novell is the first company out of the gate to interoperate with Google Wave. Meet Novell Pulse. Like wave, it'll be a cloud-hosted service for enterprise collaboration at first, but will come out in a software package later. Not at all surprisingly, this will be a commercial product Novell will attempt to make bank with. Also, it is not open source. Unlike Google, Novell makes its money from subscription costs not advertising.

That said, with Pulse offering interoperability with Wave, it is entirely possible for extra-organizational users to collaborate with in-organization users on specific items. Sort of an Open-ID enabled version of SharePoint perhaps. This could be good.

To ship or not to ship

2009-11-03T16:28:00.002-08:00

The openSUSE project is attempting a vote to determine if 11.2 is baked enough to ship it now, or if it needs to slip.

https://features.opensuse.org/308284

If you have an opinion, go ahead and vote. Or just read the comments!

Yes, there are bugs. Perhaps a lot of them. If some of these are the type to break your install, start working on it!

A matter of policy

2009-10-29T13:02:00.004-07:00

This has been a long standing policy in Technical Services, dating to the previous VP-IT and endorsed by the current one. This policy concerns email like this, generally from a manager of some kind:

"[Person X] no longer works here. Please change their password and give it to [Person Y] so they can handle email. And please set an out-of-office rule notifiying people of [Person X's] absence."

To which we politely decline. What we will do is set the out-of-office rule, that's just fine. We'll also either give a PST extract of Person X's mailbox, or if there really is no other way (the person was the Coordinator of the Z's for 20+ years and handled all the communications themselves before retiring/dying) we'll grant read-access to the mailbox to another person, and effectively turn the Person X account into a group account but lacking send-as rights.

What we will categorically not do is change a password for an inactive user and give the login to someone else. It comes down to identity theft. If we give Person Y the login info for Person X, Person Y can send email impersonating Person X. And that is wrong on a number of levels.

We resist giving access to the mailbox as well, since a non-trivial proportion of end-users give their work email as the email address for web-registration pages all over the internet. And thus that's where the "password reminder" emails get sent. Having access to someone else's mailbox is a good way to start the process of hacking an identity.

Yes, we do occasionally get a high level manager pushing us on this. But once we explain our rationalle, they've backed down so far. There is a reason we say no when we say no.

Filesystem drop-boxes on NTFS

2009-10-28T14:43:00.004-07:00

We have a need to provide dropboxes on our file-servers. Some professors don't find Blackboard's dropbox functionality meets their needs, so they rock it 1990's style. On NetWare/OES, this is a simple thing. Take this directory structure:

CLASS1:\CAS\Physics\PHYS-1234

And a group called PHYS-1234.CLASSES.WWU

Under NetWare, you set an Inherited rights filter or explicitly remove inherited rights, grant the PHYS-1245.CLASSES.WWU group the "C" trustee to the directory, and the professor's user object full rights to it. This allows students to copy files into the directory, but not see anything. On the day the assignment is due, the professor revokes the class-group's rights and tada. A classic dropbox. Dead simple, and we've probably been doing it this way since 1996 if not earlier.

It's not so simple on Windows.

First of all, Windows has different rights for Directories and Files. They use the same bits, but the bits mean different things for files and directories. For instance, one bit means both "write files" for directories, allowing users with this right to create files in the directory (analogus to the "C" NSS trustee right), and "write data" which grants the ability for a user to modify data in a file (analogus to the "M" NSS trustee right). So, this bit on a Directory grants Create, but this bit on a file grants Modify. Right.

To create a dropbox on NTFS, several things need to happen:

Inherited rights need to be copied to the directory, and inheritence blocked. (There is no Inherited Rights Filter on NTFS)
Extranious rights need to be deleted from the directory. (again with the no IRFs)
The class group needs to be granted the 'Read' rights suite to "This Folder Only", as well as "Create files".

Traverse Folder
List Folder
Read Attributes
Read Extended Attributes
Read Permissions

"CREATOR OWNER" (a.k.a. S-1-3-0) needs to be granted the 'Read' rights suite to "Subfolders and files only"

The key thing to remember here is that "Subfolders and files only" is an inheritance setting, where "This Folder Only" is a direct rights grant. Files created in this directory will get the rights defined under 'creator owner'. If the professor wishes to remove student visibility to their files, they'll have to Take Owner each file. I have found that Windows Explorer really, really likes being able to View files it just wrote, and this rights config allows that.

This series of actions will create a drop box in which students can then copy their files and still see them, but then can't do anything with it. This is because Delete is a separate right that is not being granted, and the users are not getting the "Write Data" right either. Once the file is in the directory, it is stuck as far as that user's concerned. If a user attempts to save over the invisible file of another user, perhaps the file names are predictable, they'll get access-denied since they don't have Write Data or Delete to that invisible file.

If you're scripting this, and for this kind of operation I strongly recommend it, use icacls. It'd look something like this:

icacls PHYS-1234 /inheritance:d
icacls PHYS-1234 /remove CAS-Section
icacls PHYS-1234 /grant Classes-PHYS-1234:(rx,wd)
icacls PHYS-1234 /grant ProfessorSmith:(M)
icacls PHYS-1234 /grant *S-1-3-0:(oi)(ci)(rx)

(rx,wd) = Read-Execute & Write-Data
(M) = The "Modify" simple right. Essentially Read/Write without access-control.
(oi) = Object-Inherit, a.k.a. Files
(ci) = Container-Inherit, a.k.a. Directories
(rx) = Read-Execute
*S-1-3-0 = The SID of "CREATOR OWNER". An explicit grant to this SID works better than using the name, in my experience.

This hasn't been battle tested yet, but it seems to work from my pounding on it.

You can tell I've been at this a while

2009-10-28T09:26:00.003-07:00

Last night while I was sleeping, I had a dream. In my dream I was at my desk at work. I picked up my flashlight for some reason and just then the power decided to drop. DARKNESS. And the UPS alarm in the distance. This was concerning since my workstation is on a power outlet attached to the datacenter UPS, so if my computer was out, chances were real good the entire datacenter was also down. Very bad.

Happily I just happened to have my flashlight in hand! So I powered on and went to the datacenter door. But my access card wouldn't work. The card-reader has its own internal battery, so it not reading me at all, or even giving me the access-denied angry-beep, was doubly bad. Happily, coworker dropped by and could get in so I ghosted on in behind him. The room was noisy and had all the right lights. But the UPS was still alarming. Not surprising, it's supposed to do that.

Then I woke up. I checked the clock, still had power. And there was a beep in the distance.

A smoke alarm was crying for a new battery. At 5:30am. It's just a single beep, but it seems my unconscious mind interpreted that as a UPS alarm even though those are ususally three beeps.

I have a new boss

2009-10-27T08:34:00.001-07:00

And he'll be my boss on the 16th of November.

More info once I know when it's safe to disseminate it.

Insecure applications

2009-10-23T10:55:00.004-07:00

Anyone who deals with network security has run into this problem:

Department/powerful-user buys an application for a lot of money. They would like it to work please. Application's requirement state, "disable all security systems so our crappy-app can work unencumbered." Crappy-app runs into network security problems and dies. Department/PU contacts IT and asks to have network security disabled so their expensive crappy-app can run correctly.

What happens next is a very good test of management's commitment to network security. Will management say:

Hmm, that's a lot of money. IT, make an exception for this app.
Hmm, that's a lot of money. We'll have to make it work somehow.
That's a really insecure app, too bad you spent a lot of money. It will not be installed. Let this be an object lesson to you all.

We just got a request for something like this. Apparently the application's requirements include disabling the Windows firewall. We've turned it on by GPO, so it will always be on. This is the secure way to live. Whether or not we get told to make an exception, make it work somehow, or ignore it remains to be seen.

Windows 7 releases!

2009-10-22T09:13:00.003-07:00

Or rather, its retail availability is today. We're on a Microsoft agreement, so we've had it since late August. And boy do I know that. I've been having a trickle of calls and emails ever since the beta released about various ways Win7 isn't working in my environment and whether I have any thoughts about that. Well, I do. As a matter of fact, Technical Services and ATUS both have thoughts on that:

Don't use it yet. We're not ready. Things will break. Don't call us when it does.

But as with any brand new technology there is demand. Couple that with the loose 'corporate controls' inherent in a public Higher Ed institution and we have it coming in anyway. And I get calls when people can't get to stuff.

The main generator of calls is our replacement of the Novell Login Script. I've spoken about how we feel about our login script in the past. Back on July 9, 2004 I had a long article about that. The environment has changed, but it still largely stands. Microsoft doesn't have a built in login script the same way NetWare/OES has had since the 80's, but there are hooks we can leverage. One of my co-workers has built a cunning .VBS file that we're using for our login script, and does the kinds of things we need out of a login script:

Run a series of small applications we need to run, which drive the password change notification process among other things.
Maps drives based on group membership.
Maps home directories.
Allows shelling out to other scripts, which allows less privileged people to manage scripts for their own users.

A fair amount of engineering did go into that script, but it works. Mostly. And that's the problem. It works good enough that at least one department on campus decided to put Vista in their one computer lab and rely on this script to get drive mappings. So I got calls shortly after quarter-start to the effect of, "your script don't work, how can this be fixed." To which my reply was (summarized), "You're on Vista and we told y'all not to do that. This isn't working because of XYZ, you'll have to live with it." And they have, for which I am greatful.

Which brings me to XYZ and Win7.

The main incompatibility has to do with the NetWare CIFS stack. Which I describe here. The NetWare CIFS stack doesn't speak NTLMv2, only LM and NTLM. In this instance, it makes it similar to much older Samba versions. This conflicts with Vista and Windows 7, which both default their LAN Manager Authentication Level to "NTLMv2 Responses Only." Which means that out of the box both Vista and Win7 will require changes to talk to our NetWare servers at all. This is fine, so long as they're domained we've set a Group Policy to change that level down to something the NetWare servers speak.

That's not all of it, though. Windows 7 introduced some changes into the SMB/CIFS stack that make talking to NetWare a bit less of a sure thing even with the LAN Man Auth level set right. Perhaps this is SMB2 negotiations getting in the way. I don't know. But for whatever reason, the NetWare CIFS stack and Win7 don't get along as well as the Vista's SMB/CIFS stack did.

The main effect of this is that the user's home-directory will fail to mount a lot more often on Win7 than on Vista. Also, other static drive mappings will fail more often. It is reasons like these that we are not recommending removing the Novell Client and relying on our still in testing Windows Login Script.

That said, I can understand why people are relying on the crufty script rather than the just-works Novell Login Script. Due to how our environment works, The Vista/Win7 Novell Client is dog slow. Annoyingly slow. So annoyingly slow that not getting some drives when you log in is preferable to dealing with it.

This will all change once we move the main file-serving cluster to Windows 2008. At that point, the Windows script should Just Work (tm). At that point, getting rid of the Novell Client will allow a more functional environment. We are not at that point yet.

It's the little things

2009-10-15T14:07:00.003-07:00

Right now our Microsoft migration schedule is hung up on backup licenses. Backing up clustered servers requires extensions, which we didn't notice back when we priced out the project. It is things like these that make for cost-overruns. The long and the short of it is, we're not migrating anything until we can legally back up the new environment. Period. That's just how it is.

As most of the budget arm-wrestling happens above me, I only get bits and pieces. Since we don't spend our money, we spend other people's money, we have to convince other people that this money needs to be spent. I understand there was some pushback when the quote came in, and we've been educating about what exactly it would mean if we don't do this.

I understand the order is in the works, and we're just waiting on license codes. But until they arrive (electronic delivery? What's dat?) we simply can not move forward. That's just how it is.

Clearly I am missing something

2009-10-15T07:33:00.003-07:00

On the opensuse-factory list this exchange has happened several times:

Q: Installation from LiveDVD is broken. Bug?

A: LiveCD's are not installation sources.

Clearly, something has changed in the Land of Linux Installers. Enough mind-share has shifted to "I install my linux with my LiveDVD" that it has become a very common question on the factory list when it doesn't work. When I was a kid, we installed our linux from an Install CD. LiveCD's were for things like Knoppix, used for ass-saving or quick access to Linux tools that don't exist on Windows. I seem to remember a way to install Knoppix to a hard-drive, but I never did so.

When did this change? Is this something Ubuntu is doing?

BrainShare returns for 2010?

2009-10-06T12:12:00.003-07:00

Novell just posted the Call For Participation, essentially soliciting session proposals, for BrainShare 2010. So it sounds like they're at least planning on going for it for 2010. Obviously, what with this little project I'm working on I won't be going. But it is nice to see it up and running.

Posting will be light. I was out sick last week, and I have family arriving later this week and in to next week.

I have a degree in this stuff

2009-09-30T16:32:00.002-07:00

I have a CompSci degree. This qualified me for two things:

A career in academics
A career in programming

You'll note that Systems Administration is not on that list. My degree has helped my career by getting me past the "4 year degree in a related field" requirement of jobs like mine. An MIS degree would be more appropriate, but there were very few of those back when I graduated. It has indirectly helped me in troubleshooting, as I have a much better foundation about how the internals work than your average computer mechanic.

Anyway. Every so often I stumble across something that causes me to go Ooo! ooo! over the sheer computer science of it. Yesterday I stumbled across Barrelfish, and this paper. If I weren't sick today I'd have finished it, but even as far as I've gotten into it I can see the implications of what they're trying to do.

The core concept behind the Barrelfish operating system is to assume that each computing core does not share memory and has access to some kind of message passing architecture. This has the side effect of having each computing core running its own kernel, which is why they're calling Barrelfish a 'multikernel operating system'. In essence, they're treating the insides of your computer like the distributed network that it is, and using already existing distributed computing methods to improve it. The type of multi-core we're doing now, SMP, ccNUMA, uses shared memory techniques rather than message passing, and it seems that this doesn't scale as far as message passing does once core counts go higher.

They go into a lot more detail in the paper about why this is. A big one is hetergenaity of CPU architectures out there in the marketplace, and they're not just talking just AMD vs Intel vs CUDA, this is also Core vs Core2 vs Nehalem. This heterogenaity in the marketplace makes it very hard for a traditional Operating System to be optimized for a specific platform.

A multikernel OS would use a discrete kernel for each microarcitecture. These kernels would communicate with each other using OS-standardized message passing protocols. On top of these microkernels would be created the abstraction called an Operating System upon which applications would run. Due to the modularity at the base of it, it would take much less effort to provide an optimized microkernel for a new microarcitecture.

The use of message passing is very interesting to me. Back in college, parallel computing was my main focus. I ended up not pursuing that area of study in large part because I was a strictly C student in math, parallel computing was a largely academic endeavor when I graduated, and you needed to be at least a B student in math to hack it in grad school. It still fired my imagination, and there was squee when the Pentium Pro was released and you could do 2 CPU multiprocessing.

In my Databases class, we were tasked with creating a database-like thingy in code and to write a paper on it. It was up to us what we did with it. Having just finished my Parallel Computing class, I decided to investigate distributed databases. So I exercised the PVM extensions we had on our compilers thanks to that class. I then used the six Unix machines I had access to at the time to create a 6-node distributed database. I used statically defined tables and queries since I didn't have time to build a table parser or query processor and needed to get it working so I could do some tests on how optimization of table positioning impacted performance.

Looking back on it 14 years later (eek) I can see some serious faults about my implementation. But then, I've spent the last... 12 years working with a distributed database in the form of Novell's NDS and later eDirectory. At the time I was doing this project, Novell was actively developing the first version of NDS. They had some problems with their implementation too.

My results were decidedly inconclusive. There was a noise factor in my data that I was not able to isolate and managed to drown out what differences there were between my optimized and non-optimized runs (in hindsight I needed larger tables by an order of magnitude or more). My analysis paper was largely an admission of failure. So when I got an A on the project I was confused enough I went to the professor and asked how this was possible. His response?

"Once I realized you got it working at all, that's when you earned the A. At that point the paper didn't matter."

Dude. PVM is a message passing architecture, like most distributed systems. So yes, distributed systems are my thing. And they're talking about doing this on the motherboard! How cool is that?

Both Linux and Windows are adopting more message-passing architectures in their internal structures, as they scale better on highly parallel systems. In Linux this involved reducing the use of the Big Kernel Lock in anything possible, as invoking the BKL forces the kernel into single-threaded mode and that's not a good thing with, say, 16 cores. Windows 7 involves similar improvements. As more and more cores sneak into everyday computers, this becomes more of a problem.

An operating system working without the assumption of shared memory is a very different critter. Operating state has to be replicated to each core to facilitate correct functioning, you can't rely on a common memory address to handle this. It seems that the form of this state is key to performance, and is very sensitive to microarchitecture changes. What was good on a P4, may suck a lot on a Phenom II. The use of a per-core kernel allows the optimal structure to be used on each core, with changes replicated rather than shared which improves performance. More importantly, it'll still be performant 5 years after release assuming regular per-core kernel updates.

You'd also be able to use the 1.75GB of GDDR3 on your GeForce 295 as part of the operating system if you really wanted to! And some might.

I'd burble further, but I'm sick so not thinking straight. Definitely food for thought!

Browser usage on tech-blogs

2009-09-28T11:16:00.004-07:00

Ars Technica just posted their August browser update. They also included their own browser breakdown. ArsTechnica is a techie site, so it comes as no surprise what so ever that Firefox dominates at 45% of browser-share. This made me think about my own readership.

As you can see, Firefox makes up even more of the browser-share here (50.34%). Interestingly on the low end, Opera is actually the #3 browser (4.46%), not Safari (3.43%). Looking at the version breakdown for those IE users, only 17% of them are on IE6. Yay!

ArsTechnica's Safari numbers are not at all surprising, since they cover a fair amount of Apple news and I don't.

So yeah, Tech blogs and sites don't have a lot of IE traffic. Or, so I believe. What are your numbers?

More thoughts on the Novell support change

2009-09-25T15:00:00.003-07:00

Something struck me in comments on the last post about this that I think needs repeating on a full post.

Novell spent quite a bit of time attempting to build up their 'community' forums for peer-support. Even going so far as to seed the community with supported 'sysops' who helped catalyze others into participating, and creating a vibrant peer support community. This made sense because it built both goodwill and brand loyalty, but also reduced the cost-center known as 'support'. All those volunteers were taking the minor-issue load off of the call-in support! Money saved!

Fast forward several years. Novell bought SuSE and got heavily into Open Source. Gradually, as the OSS products started to take off commercially, the support contracts became the main money maker instead of product licenses. Just as suddenly, this vibrant goodwill-generating peer-support community is taking vital business away from the revenue-stream known as 'support'. Money lost!

Just a simple shift in the perception of where 'support' fits in the overall cost/revenue stream makes this move make complete sense.

Novell will absolutely be keeping the peer support forums going because they do provide a nice goodwill bonus to those too cheap to pay for support. However.... with 'general support' product-patches going behind a pay-wall, the utility of those forums decreases somewhat. Not all questions, or even most of them for that matter, require patches. But anyone who has called in for support knows the first question to be asked is, "are you on the latest code," and that applies to forum posts as well.

Being unable to get at the latest code for your product version means that the support forum volunteers will have to troubleshoot your problem based on code they may already be well past, or not have had recent experience with. This will necessarily degrade their accuracy, and therefore the quality of the peer support offered. This will actively hurt the utility of the peer-support forums. Unfortunately, this is as designed.

For users of Novell's active-development but severe underdog products such as GroupWise, OES2, and Teaming+Conferencing, the added cost of paying for a maintenance/support contract can be used by internal advocates of Exchange, Windows, and SharePoint as evidence that it is time to jump ship. For users of Novell's industry-leading products such as Novell Identity Management, it will do exactly as designed and force these people into maintaining maintenance contracts.

The problem Novell is trying to address are the kinds of companies that only buy product licenses when they need to upgrade, and don't bother with maintenance unless they're very sure that a software upgrade will fall within the maintenance period. I know many past and present Novell shops who pay for their software this way. It has its disadvantages because it requires convincing upper management to fork over big bucks every two to five years, and you have to justify Novell's existence every time. The requirement to have a maintenance contract in order for your highly skilled staff to get at TIDs and patches, something that used to be both free and very effective, is a real-world major added expense.

This is the kind of thing that can catalyze migration events. A certain percentage will pony up and pay for support every year, and grumble about it. Others, who have been lukewarm towards Novell for some time due adherence to the underdog products, may take it as the sign needed to ditch these products and go for the industry leader instead.

This move will hurt their underdog-product market-share more than it will their mid-market and top-market products.

If you've read Novell financial statements in the past few years you will have noticed that they're making a lot more money on 'subscriptions' these days. This is intentional. They, like most of the industry right now, don't want you to buy your software in episodic bursts every couple years. They want you to put a yearly line-item in your budget that reads, "Send money to Novell," that you forget about because it is always there. These are the subscriptions, and they're the wave of the future!

Very handy but terrible plugin

2009-09-24T18:44:00.003-07:00

Yes, this plugin is a terrible idea.

But then, so are appliances with built in self-signed SSL certificates you can't change. You take what you can get.

Novell Support: Now even MORE behind a pay-wall!

2009-09-23T12:55:00.001-07:00

I first ran into this on Bucky's Blog. Specifically, Novell is changing what non-paying users can get out of Novell's support options. The details are still being hashed out, but they made the mistake of running afoul of one of the major no-no's of support: Pay-for-patches, or at least the suggestion of it. They caught a lot of flack about that with requiring a support contract to use the auto-update channels for their Linux products, but this will go even farther and put even support packs behind the maintenance-contract pay-wall.

So if you're a NetWare customer that hasn't paid maintenance in umpteen years since your server Just Works (TM), you'll now have to buy maintenance if you want to apply the latest Service Pack. Or if your server is throwing abends that can be fixed with a patch that you learned about in the peer support forums, you'll need a contract to be able to access it. This was done intentionally to pull in these free-loaders into paid support, but it does represent a potentially steep cost that can catalyze more migrations off of Novell products. This will hurt the shoe-string IT departments more than the big-bucks one. And since that describes a goodly percentage of 'small businesses' this could be a major problem in the future.

What's causing some confusion is their intent to put some of the KB articles behind the pay-wall as well. As described by Novell's support-community coordinator:

FACT: Only about 8% of the TIDs in the knowledgebase will be closed off
for entitled customers. Those are the TIDS for the products under "General
Support" ( http://support.novell.com/lifecycle ). All other TIDS will
remain open to the general public. As products move from general support
to extended and self support, all TIDS will become public.

So the 20+ year history of NetWare TIDs will still be there as NetWare is nolonger on general support per-se, but TIDs about currently in support closed-source items like Novell Identity Manager and the entire ZenWorks line is another story. One beef I have about this is that even if you do have a maintenance contract, it means that anyone who could possibly search the KB for articles has to have:

A novell.com login
Their novell.com login associated with a maintenance contract

This doesn't always happen. I've had to add a few people to our contract so they can use the Customer Center to get license codes or register SLES machines against our support. But the large majority of our historic NetWare admins aren't on the contract because they haven't needed it. This move will force organizations such as ours to much more actively manage our Customer Center contract/username associations. That can be a lot of bother.

The end effect of all of this is that the value of 'peer support' is markedly reduced for currently-shipping products. Once upon a time Novell was a company that really encouraged peer support since it took load off of their support engineers, customers liked it since it was free, and it encouraged quite a lot of goodwill. Now they seem to have realized that this was a drain on the bottom line and are dismantling the system in favor of everyone paying for support. This destroys goodwill, as they're now learning in the support forums.

Printing habits

2009-09-21T14:52:00.004-07:00

Some students are going to be in for a rude, rude surprise real soon. Today alone there is a student who has printed off 210 pages. Looking at their print history, they printed off 100 copies of two specific handouts (in batches of 50), and that's 40% of their entire quota for the quarter. Once they hit the ceiling, they'll have to pay to get more. This is different from last year!

We always got a few students who rammed their head against the 500 page limit within two weeks of quarter start. I'm sure we'll get some this quarter too. There may be heated tempers at the Helpdesk as a result, but thems the breaks.

Quarter start: printing

2009-09-21T13:15:00.001-07:00

Today is go-live for the new Microsoft/PCounter based printing system. It hasn't gone off perfectly, but most of the problems so far have been manageable. Also, it's only Monday. The true peak load for printing will be Wednesday between 11:00 and 12:00. Wednesday is when classes start.

So far the big problem is that some of the disk images used for the labs included printers they weren't supposed to, a side effect of how Microsoft does printing. All in all, it's a pretty small thing but it does ruin the clean look. The time between when Summer session stopped and when all the images had to be applied (last Friday) was the same time we get every year, but this year included major changes we haven't seen since we converted from queue-based printing to NDPS printing back around 2002. So yeah, these kinds of QA things can get dropped when under this kind of time pressure, and just plain new environment.

Also, the Library doesn't have their release stations up yet. They'll have them there by the end of the day, but the fact remains that they're on the old system until then. Due to the realities of accounting, each student was given only 50 pages this morning on the old system. Which means that some users are already whacking their heads on the limit. They'll have to go to one of the ATUS labs to print, as they're all on the new system and their quotas are much higher there. If Libraries doesn't have it by tomorrow, something will have to give.

The end of the line for RAID?

2009-09-18T12:27:00.003-07:00

Regarding this: http://www.enterprisestorageforum.com/technology/features/article.php/3839636

He has a point. Storage sizes are increasing faster than reliability figures, and the combination is a very bad thing for parity RAID. Size by itself means that large RAID sets will take a long time to rebuild. I ran into this directly with the MSA1500 I was working with a while back, where it would take a week (7 whole days!) to rework a 7TB disk-array. The same firmware also very strongly recommended against RAID5 LUNs on more than 7TB of of disks due to the non-recoverable read error rate on the SATA drives being used. RAID6 increase the durability of parity RAID, but at the cost of increased overhead.

Unfortunately, there are no clear answers. What you need really depends on what you're using it for. For very high performance storage where random I/O latency during high speed transfers are your prime performance metric, lots of cheap-ass SATA drives on randomized RAID1 pairs will probably not be enough to keep up. Data-retention archives where sequential write speeds are your prime metric is more forgiving and can take a much different storage architecture, even though it may involved an order of magnitude more space than the first option here.

One comment deserves attention, though:

The fact is that 20 years ago, a large chunk of storage was a 300MB ESDI drive for $1500, but now a large drive is hard to find above $200.

Well, for large hard drives that may be true but for medium size drives I can show you many options that break the $200 barrier. 450GB FC drives? Over $200 by quite a lot. Anything SSD-Enterprise? Over $200 by a lot, and the 'large drive' segment is at an order of magnitude over that.

We're going to see some interesting storage architectures in the near future. That much is for sure.