Main Site News at A.P.Lawrence.com

A strangely compromised Linux box

Linux,Security 2009/11/05

A customer reported that a Linux machine used for ssh access (to in turn give telnet access to an ancient SCO machine) was refusing logins. I asked him to try logging in as root at the console; he was unable to do so.

When I arrived on site, I found that I could not login as he had said. I rebooted to single use mode and started peeking around. The machine had been hacked; there was little doubt about that. It's HOW it was hacked that bothers me,

First, there was no attempt to hide any evidence. I could see in wtmp and the secure logs that someone had logged in from a German ISP address, attained su status, and created a new su user for himself. He then changed root's password.

Fine so far, right? But then he did something very strange. He hand edited /etc/passwd and added "/nologin" at the end of each line except root and his own. This was what was preventing people from logging in.

Why do that?

My first thought was that this was just a disgruntled employee doing minor mischief. But when I went multi-user and started checking more, I found this:

COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
3       2614 root    3u  IPv4   8033       TCP *:ircd (LISTEN)

That looks like the machine has been put into a botnet. I ran rkhunter but didn't find anything else unusual.

This is very odd. If you want the machine for a botnet, why disable the user logins, which only serves to immediately call attention to the machine?

Another oddity: this same issue happened several months earlier. That is, users could not login and the root password was changed. That time, the user access came back before I could get there and I had them boot to single user mode to change the root password. I wish I knew if an irc daemon was running then, but I attributed all of that to user error or a router glitch.

Could it be just an inept hacker? A "kiddie script" that disables logins? But why undo its work? And why redo it now?

And he DID redo it. The time stamps are plain: he did all this just days ago. It makes no sense.

I suspect that this person got in because someone's home machine is already part of the botnet. I don't know how he attained escalated permission, but once you have physical access, all bets are off. We'll have to reinstall the machine, but if I can't identify the source, what's the point?

I don't know. I'm really not sure what to do. For the moment, I've locked down ssh so that only I can get on - I want to see if he does have another back door. But I'm also concerned about other machines in the network - any of these could be compromised also. So where do we go from here? I don't want to put this customer to a lot of expense for nothing, but the whole situation is disquieting.

It does offer a lesson though: when something odd like that happens, we should take the time to look more deeply. If I had spotted that ircd months ago, I'd have... what? I don't know. But still, I should have looked deeper then.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. If you have any question, please do feel free to contact me.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

When Samba Pigs Fly

Troubleshooting,Linux,Samba 2009/11/04

Yesterday a customer called because he needed to be able to write into a certain share on his Samba server. I ssh'd right in, made the change to the config file, restarted Samba and shot him off an "All set!" email.

Such confidence had I that immediately after hitting send, I left my house to do some errands and when I realized I had forgotten my phone, I didn't even bother to go back for it: everything is under control, all pigs are fed and ready to fly.

Yeah. When I got home, I found both phone and email messages from my customer. Such a nice guy he is - he was APOLOGIZING to me because it didn't work. "Maybe I'm doing something wrong?", he asked.

I ssh'ed in again and saw my "mistake". I had written "writeable" rather than "writable" in the config file. I quickly fixed that, restarted Samba, snapped off another email explaining my error and took a break for lunch.

Unfortunately the pigs seemed to still be having a little trouble with the flying stuff. I had barely bitten into my sandwich before he called again. Permission denied. Can't do it. Was he doing something wrong?, he begged to know? Of course not, I assured him. The damn pigs were just being stubborn.

I double checked. Yes, he had write permission in the directory. What the heck? Here's part of the config file for your amusement:

[homes]
	comment = Home Directories
	read only = No
	browseable = No

[printers]
	comment = All Printers
	path = /var/spool/samba
	printable = Yes
	browseable = No


[Syn75]
	comment = syn75
	path = /usr/syn75
	browseable = yes
            read only= Yes

[CPONLINE]
	comment = cponline
	path = /usr/syn75/00/CPONLINE
	browseable = yes
	writable = Yes

Those pigs have wings, dammit! So exactly what happens, I asked?

"I choose Save As. I navigate down to CPONLINE..."

Ooops. Magic word. He said "Down", didn't he? The pigs all perked up and started tentatively fluttering their wings. I asked the $64,000 question: "Are you going through the Syn75 share or the CPONLINE share?"

NO, he was not using CPONLINE. He was navigating down through the Syn75 share. THAT share has no write permission - it doesn't matter that CPONLINE is under it, that only is writable if you come to it through the CPONLINE share! I had him map a network drive to CPONLINE and the pigs lifted off into the sky and everybody was happy.

Because he's such a nice guy ("Maybe I'm doing something wrong?) and because I should have paid more attention when he asked that, I'm not even sending him a bill for any of it.

Pigs: to your stations! Fly, you pink porkers, fly!

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Gentle Waves

Web-HTML,Blogging,Advertising 2009/11/03

Let me first apologize to the people who don't yet have Wave accounts - it must be very boring and also frustrating listening to people wax on about things you can't yet experience.

On that note, I do have a few invitations left for regular contributors, customers, and so on.. Just drop me an email.

For those who do have accounts, I'm "pcunix@googlewave.com" and you can find my public waves by searching "with:public creator:pcunix".

Now what I really wanted to talk about.

Last night, I presented Google Wave to our little retirement community Computer Club. They are a mixed crowd: we have a guy who programmed systems in the early 50's for the Department of Defense, another man who managed programmers before he retired, other people who used computers extensively at their jobs... and people who are struggling to understand email.

I can't make everybody happy, but I do try to keep things basic enough for the newbies and deep enough not to bore the old hands. Obviously that can never be entirely successful, but I thought the reactions last night were interesting.

Some people "got it" very quickly. They understood it so much that they were arguing with the people who didn't get it, saying things, like "No, really, this would have been a fantastic tool to have for the work I used to do!" Others were obviously confused.

One man in particular almost seemed angry. "It's confusing", he said. "Why do I need all this stuff that it does? I don't - I do fine with email!"

I was momentarily tempted to ask why he comes to Computer Club if he doesn't want to learn anything new, but I realized that wasn't what he meant: he just doesn't want to learn a new way of doing email. Email as it exists now meets his needs, he doesn't need to combine it with IM, doesn't need in-line replies, doesn't need Yes/No gadgets - doesn't need the confusion, thank you very much. If Google or anybody else is going to try to drag him into using Wave, he'll be kicking and screaming all the way.

I offered my argument that as we start to use Wave for some things, we'll realize that we ought to start with a Wave just because we might need it to be a Wave later. Someone else agreed, pointing out that if you have two ways of doing something, you'll naturally settle in to using one, probably the more powerful way, even if you don't use all of its features all the time. Our Mr. Confused was having none of that, though. As I said, he seemed a little angry - perhaps he had the idea that Google was somehow going to force this upon him.

That's not how its going to happen. Nor are people like him ever going to ask for a Wave account just to kick the tires. Too confusing, no perceived value, not going to happen.

I think that what WILL happen is that Mr. Confused and Mrs. I Never Heard of It Anyway are going to get softly dragged in. That is, they'll click on a Web page that invites them to join a discussion group or to get more information about some subject they are interested in. When they access that link, they'll need a Wave account. By that time, that will be an instant or near instant process - much like getting a Gmail account now. To these people, this will just be another web page, something they need this "Wave account" to access, but that's simple.

They may not even realize at first that they have a new tool. It's just a discussion about local tax rates with their neighbors, or a list of resources and information about something else. They may not realize that they now have the ability to create their own waves. They may know nothing about Yes/No gadgets until they see one and use it. But over time, as they have joined more Waves, they'll start to "get it". Maybe a more technical friend will show them a few tricks, maybe they'll read a little how-to at another web page - or at another Wave!

That's probably how it's going to happen for a lot of people. No kicking and screaming, just a gentle slide into something new.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Misunderstanding Wave

Web-HTML,Blogging,Advertising 2009/11/02

There are people who don't understand Google Wave. There are people who don't like Google Wave. There are people who do understand Wave, and people who do like it. Most of those who don't like it just don't understand it, but even a few who do understand still don't like it.

Most (maybe even all) of the complaints you'll hear about Wave are gripes about problems that obviously will be fixed as this progresses out if its current "Preview" stage. Most of the complaints are also client-side implementation issues that have nothing whatsoever to do with the underlying concepts. You need to keep that in mind when listening to negative comments.

One of the most important things to understand about Wave is that anyone can create a Wave server or a Wave client. You can go to http://google.com/wave to use Google's web based client, but I and many other Mac users use Waveboard, a third party Wave client. If you Google for "google wave client", you'll find many other clients. Doing a search for "google wave servers" doesn't yield quite so much, but I did spot at least one, and as time goes on there will be more. Although I and others often refer to "Google Wave", in fact the idea is that Wave servers will be like SMTP servers: anybody can run one and your Wave server will happily talk to any other Wave server.

Let's have a look-see at some of the griping.

Interactive Chat is distracting

Google Wave is IM squared. Not only do you see what the other person is typing, but if there are many people involved in the Wave, you see all of them typing, back-spacing, correcting things - the screen jumps around and it can all be very annoying.

The biggest problem there is the jittery screen. That is, of course, a client side issue - nothing says you HAVE to have the screen updated in real time. And nothing says you have to participate in a Wave that has dozens of people actively typing. One of the truly beautiful things about Wave is the "replay" ability - you can come back when all the excitement has died down and run through the whole thing step by step at your own pace.

I have confidence that client-side issues like this will be fixed, and soon.

It's sloooowwww

If a lot of people are in a Wave, it does get slow. That feels like a client side issue to me - just don't try to show all that activity at the same time. Buffer it up and display it when things calm down a bit.

Big Waves Break

Early adopters are finding that large Waves crash and burn. It's not hard to split off and start another Wave, but that needs to be fixed. I don't know if that's client side, server side or a general weakness in the protocol, but it needs fixing. Again: Preview release.

Difficult Contact Management

When I first got Wave, I found my contacts list populated with people who already have Wave accounts. These people were apparently people I know, or at least have had email correspondence with. I use Gmail, so Google probably pulled them from there.

I don't recognize half the people on that list.

The reason is simple enough: their Wave account doesn't match whatever I know them as. Google knows them, and knows the connection between that account and whatever email I know them as, but Google doesn't let me see that connection.

Presently, you can't organize your Wave contacts into groups. Obviously that's a necessary and useful feature and justas obviously it WILL be added. But right now? Nope.

Broken features

Remember, this is a "Preview". Sometimes things that are supposed to work get balky. Sometimes your Client loses contact with its server. Sometimes just plain weird stuff happens. For example, I had marked a Wave as "public", which means that anyone can see it and add to it. The darn thing kept losing its public status. It seems to be OK now, but that's annoying.

You can find my public waves by searching (in Wave) for "with:public creator:pcunix"

Spammers

It's unclear how the problem of misbehaving people will be dealt with. Right now, if you add someone to a Wave, you can't take them off easily or even just block their messages from your view. In a public Wave, any idiot can join the conversation and you can't filter them out. People can add objectionable 'bots to your Waves - somebody added Eliza Robot to one of my public Waves. I was able to delete that, but this kind of nonsense does happen and we will need ways to prevent it.

Unwanted Invitations

The matter of you being added to Waves you don't want to be part of is a common complaint, but there's a simple fix - just "mute" the Wave and it won't bother you again. The mute function moves the Wave out of your inbox and ignores any updates that would bring it back to your attention. If you ever change your mind, you can drag it back to your inbox, but otherwise it has been gagged and silenced.

Just don't understand it

I've talked to people who think Wave is IM. Others think it is email. Still others think it's a Wiki.

It is all of that, and more. Some people, stuck in their false perceptions, may never see the reality. I think as more of us start using Wave, the confused nay-sayers like Robert Scoble will eventually understand.

Just this morning one of my Wave contacts posted a new Wave about using a Wave as a Technical information log. He says:

Trying to visualize how the different elements stitch together is often almost as hard as starting from scratch. The more I play with waves the more it really seems like a one stop shop for interfacing to information.

He has the right idea. Robert Scoble may not understand yet, but others do. The Waves are coming!

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

My first Waves

Web-HTML,Blogging,Advertising 2009/10/31

Thanks to one of our readers (thanks again, Donal) I got a Google Wave invitation Friday morning. By the way - if someone says they have sent you an invitation, you may have to wait: Donal said that he had sent that invitation on Monday. I don't know if Google is just slow in processing these or if they are deliberately doling them out slowly (probably the latter), but once you actually get your invite, you can be up and working in minutes.

I started out using Wave in my Firefox browser, but quickly switched to Waveboard, a Mac Wave client. It's not that it's all that much better than running Wave in a browser; it's just that I like having it in its own Dock icon.

I created a few waves and soon had a few conversations going with other Wave users who I added to the Waves. My contacts as supplied by Google seem to be people from my Gmail contacts who also have Wave accounts. I recognize only about half of them, though: probably because they used a different name in email than they do in Wave.

My first conversation was with Donal, thanking him for the invitation. That could have just as easily been done in email or chat, of course; there was no specific reason to use Wave. However, in the next conversation, Wave was useful.

I had started a Wave titled "Until everyone can use this, sure is useless :-)", in which I lamented my inability to bring in people who don't currently have Wave accounts. I have uses in mind for Wave, but without being able to add in non-wave users, I can't do anything useful. I added in everyone in my Contacts list and a few comments soon came.

One of the people happened to be someone I do business with and his appearance reminded me that there was something I wanted to talk to him about. We started doing that in a "private" conversation within the existing wave, but then realized that it was better to spawn it off to a new Wave. That's very easy to do and is an advantage of Wave over Mail and Chat - not that you can't peel off from either, but it's easier in Wave.

I also started a "public" Wave. That's a Wave that anyone can join (assuming you have already been blessed with a Wave account). You create a public Wave by adding "public@a.googlewave.com" to the list of people you want to be able to read the Wave. With that, it's now open to the world. Presently, there is no way to post a link to a public Wave; you have to search within Wave to find them. For example, to find my Wave, you'd search for "with:public Tony Lawrence's Unix, Linux and Mac OS X Tips".

Click for larger image

Note that it's perfectly possible to insert advertising into your Waves. I did that here with simple text links, but you could put in Javascript with a Wave Gadget. How long before there's an Adsense Gadget? I'd guess not very long.

I started another Wave called "Will Wave replace Email?". I opined:

For SOME email conversations, Wave is much better. The problem (for mail) is that you don't necessarily know ahead of time that a conversation would be better in Wave. So... once this is ubiquitous and we all understand that, we might just start using Wave instead of email at the beginning of a conversation?

Two people have commented on that so far. One said:

If Google is going to succeed with this they're going to have to do two things.

1. Make it so user@googlewave.com is a real e-mail address that can be used by anyone. Regular e-mails sent to these addresses will automatically turn into Waves for the Wave user.

2. Make it so that a Wave user can create a wave and add people to it that use regular e-mail. Waves will get sent to standard e-mail users as regular messages and get sorted into threads the way their e-mail client chooses to do it.

I suspect that's exactly what will happen. A Wave isn't always better than email, but sometimes it is and you don't necessarily know at the beginning of a conversation that it would be better as a Wave. For example, suppose that halfway through a long back and forth email discussion you need to bring someone else in. With email, you'd have to forward all the prior messages - that can be tough for the recipient. With Wave, you just bring them in and they can replay the previous messages step by step if they want to. That's a powerful advantage over email and it's why, if this does become ubiquitous, many of us will probably just use Wave instead of email.

That could really change things, couldn't it?

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Help, I'm disappearing!

Web-HTML 2009/10/29

When we first moved here, our address didn't exist in Google Maps, nor could anything but dirt be seen in the satellite images. That changed soon enough and for some time now I've been able to plug in my home address as a starting point for directions.

A year or so ago, I started having a little trouble with that. The reason was because someone I do business with added me to a user generated map. From that point on, Google saw my address as a business address, and would react by asking "Did you mean A.P. Lawrence?". I'd just click on that and everything would be fine.

The other day I noticed that I couldn't do that. Google Maps insists that my address simply does not exist. Indeed, if I summon up a map of the town, the whole street has disappeared! It's still visible in the satellite view, as is my house and even the golf cart parked in my driveway, but the map view shows empty space.

Yahoo still knows we are here. I don't LIKE Yahoo maps, but I'm stuck with them for now. It's not all that critical; I only use these as a failsafe for my car's GPS, but it is a bit unsettling to have disappeared.

Interestingly, one of my neighbors ( who also runs a business from his home) does appear in the business listings. He's at the top of the page, even though our street no longer exists. Why is he still there but I am not? Who knows? Why does google show addresses in other States when I have specifically given both a town, a State and a zip code? How can Google have maps of a street for several years and not have them now? I do not know.

Perhaps Google knows something I do not. We are on the amorphous edges of the supposedly spooky Bridgewater Triangle ; perhaps we are slowly being sucked into another dimension? If so, it's been grand and I will write if I can.

There is a "Report a problem" link at the bottom of the Google page. I clicked on that and explained the vanishing of my street. We'll see how long it takes for Google to find us again.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Answer the bleeping email

Employment,Opinion 2009/10/28

Earlier this week I got email from VMware announcing that I could now order Fusion 3.0 for my Mac. Because I had been a beta tester of this, they offered me a coupon code that was supposed to give me a 25% discount.

To my annoyance, it did not. The order page insisted the code was invalid.

In spite of diminished income in this recession, I still have more money than patience, so I just ordered the upgrade without the discount. Of course that ticks me off, so I returned to the email that promised this boon and noted that it didn't say a word about "No reply possible", so I hit reply and (politely) expressed my disappointment.

Yeah, I know - I should not hold my breath.

I don't want to single out VMware here. Yeah, it's really dumb to send out coupon codes that you aren't honoring, but never mind that. Where VMware really fails is that I can't send them email.

That's hardly unusual. At far too many large companies today, email from outside is discouraged or blocked outright. You usually can't hit "Reply" and if you visit their web sites, you are more likely to be forced fill out contact forms that may confine you to certain subjects - your particular concern may not be among the choices.

VMware has such a system. In addition to finding nothing that matched my needs, all of their forms request extraneous information that I don't feel like providing, thank you very much anyway.

Suggestion to companies implementing such things: have a "I think you screwed me" form and DON'T have any required fields other than one of email or phone.

It's possible that someone from VMware may eventually reply, but I have little confidence of that. I could try calling them, but large company voice mail systems aren't fun to navigate. I SHOULD be able to send email. That is the most convenient way to provide everything that they'd need to either redress my complaint or tell me to go stuff it. Nobody has to write down who I am, why I got the code - it's all there, because I'm replying to their promise!

Oh, right: they'd need a lot of people to handle customer emails. Oh, boo-hoo: how much would it truly cost? How much happier would those annoying customers be if they could communicate this way?

As noted, VMware is hardly the only sinner. I'm just ticked at them because they promised me $15.00 off and didn't give it to me. I'll get over it.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

The cure for everything - chmod 777

Linux,MacOSX 2009/10/28

I admit that I have done a "chmod 777" when I should not have. Almost always that came from haste or frustration. Not frustration with Unix permissions, but frustration with whoever had daily care responsibility for the system - their inability to understand permissions might have driven me to this.

Sometimes it hardly matters. At many small businesses, everybody has "got root" anyway and has learned that this magic incantation will "fix" problems. Well, until it breaks a setuid program, of course. Nobody, NOBODY ever learns "chmod +w", do they?

Another rare breakage is /tmp. It's supposed to have the "t" bit set so that only the owners of files can delete, but I've had folks "777" it. Why? Who knows?

More usually the open permissions are applied to some common set of data. All goes well until someone removes (or just moves!) something that is needed by someone else, and then the crying starts.

The most horribly wrong things that can be done with permissions come from people who have learned about "-R" (recursive) or wild cards. Two or three times a year I find a system where someone has done a chmod to ".*". That's bad enough by itself, but when combined with -R, the results can be spectacular.

Unnecessary damage

What you should understand is that this is all unnecessary. Most of us who have to deal with the results of misunderstood chmod's wish that users didn't even know that the numeric form exists. There would be far fewer errors if users only knew the symbolic modes.

The symbolic form is also much more powerful. Consider this :

$ ls -l
total 0
-rw-r--r--  1 apl  apl  0 Oct 27 15:06 a
-rw-r--r--  1 apl  apl  0 Oct 27 15:06 b
-rwxr--r--  1 apl  apl  0 Oct 27 15:10 c

$ chmod a+X  a b c
$ ls -l
total 0
-rw-r--r--  1 apl  apl  0 Oct 27 15:06 a
-rw-r--r--  1 apl  apl  0 Oct 27 15:06 b
-rwxr-xr-x  1 apl  apl  0 Oct 27 15:06 c

ONLY the file that was already executable had full execution bits added - try THAT with Windows!

(If you did want to change all the files, you'd use "chmod a+x a b c")

But I'm being silly. People will continue to "chmod 777" anytime they have a problem. Program doesn't work? Chmod 777. Unexpected error? Chmod 777. Grinding noise inside the computer? Chmod 777. Too hot in here? Chmod 777.

Feeling frustrated by people changing permissions for no reason? By now you should know the cure. Say it with me: Chmod 777.

Don't you feel better now?

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here