Linux Site News at A.P.Lawrence.com

A strangely compromised Linux box

Linux,Security 2009/11/05

A customer reported that a Linux machine used for ssh access (to in turn give telnet access to an ancient SCO machine) was refusing logins. I asked him to try logging in as root at the console; he was unable to do so.

When I arrived on site, I found that I could not login as he had said. I rebooted to single use mode and started peeking around. The machine had been hacked; there was little doubt about that. It's HOW it was hacked that bothers me,

First, there was no attempt to hide any evidence. I could see in wtmp and the secure logs that someone had logged in from a German ISP address, attained su status, and created a new su user for himself. He then changed root's password.

Fine so far, right? But then he did something very strange. He hand edited /etc/passwd and added "/nologin" at the end of each line except root and his own. This was what was preventing people from logging in.

Why do that?

My first thought was that this was just a disgruntled employee doing minor mischief. But when I went multi-user and started checking more, I found this:

COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
3       2614 root    3u  IPv4   8033       TCP *:ircd (LISTEN)

That looks like the machine has been put into a botnet. I ran rkhunter but didn't find anything else unusual.

This is very odd. If you want the machine for a botnet, why disable the user logins, which only serves to immediately call attention to the machine?

Another oddity: this same issue happened several months earlier. That is, users could not login and the root password was changed. That time, the user access came back before I could get there and I had them boot to single user mode to change the root password. I wish I knew if an irc daemon was running then, but I attributed all of that to user error or a router glitch.

Could it be just an inept hacker? A "kiddie script" that disables logins? But why undo its work? And why redo it now?

And he DID redo it. The time stamps are plain: he did all this just days ago. It makes no sense.

I suspect that this person got in because someone's home machine is already part of the botnet. I don't know how he attained escalated permission, but once you have physical access, all bets are off. We'll have to reinstall the machine, but if I can't identify the source, what's the point?

I don't know. I'm really not sure what to do. For the moment, I've locked down ssh so that only I can get on - I want to see if he does have another back door. But I'm also concerned about other machines in the network - any of these could be compromised also. So where do we go from here? I don't want to put this customer to a lot of expense for nothing, but the whole situation is disquieting.

It does offer a lesson though: when something odd like that happens, we should take the time to look more deeply. If I had spotted that ircd months ago, I'd have... what? I don't know. But still, I should have looked deeper then.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. If you have any question, please do feel free to contact me.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

When Samba Pigs Fly

Troubleshooting,Linux,Samba 2009/11/04

Yesterday a customer called because he needed to be able to write into a certain share on his Samba server. I ssh'd right in, made the change to the config file, restarted Samba and shot him off an "All set!" email.

Such confidence had I that immediately after hitting send, I left my house to do some errands and when I realized I had forgotten my phone, I didn't even bother to go back for it: everything is under control, all pigs are fed and ready to fly.

Yeah. When I got home, I found both phone and email messages from my customer. Such a nice guy he is - he was APOLOGIZING to me because it didn't work. "Maybe I'm doing something wrong?", he asked.

I ssh'ed in again and saw my "mistake". I had written "writeable" rather than "writable" in the config file. I quickly fixed that, restarted Samba, snapped off another email explaining my error and took a break for lunch.

Unfortunately the pigs seemed to still be having a little trouble with the flying stuff. I had barely bitten into my sandwich before he called again. Permission denied. Can't do it. Was he doing something wrong?, he begged to know? Of course not, I assured him. The damn pigs were just being stubborn.

I double checked. Yes, he had write permission in the directory. What the heck? Here's part of the config file for your amusement:

[homes]
	comment = Home Directories
	read only = No
	browseable = No

[printers]
	comment = All Printers
	path = /var/spool/samba
	printable = Yes
	browseable = No


[Syn75]
	comment = syn75
	path = /usr/syn75
	browseable = yes
            read only= Yes

[CPONLINE]
	comment = cponline
	path = /usr/syn75/00/CPONLINE
	browseable = yes
	writable = Yes

Those pigs have wings, dammit! So exactly what happens, I asked?

"I choose Save As. I navigate down to CPONLINE..."

Ooops. Magic word. He said "Down", didn't he? The pigs all perked up and started tentatively fluttering their wings. I asked the $64,000 question: "Are you going through the Syn75 share or the CPONLINE share?"

NO, he was not using CPONLINE. He was navigating down through the Syn75 share. THAT share has no write permission - it doesn't matter that CPONLINE is under it, that only is writable if you come to it through the CPONLINE share! I had him map a network drive to CPONLINE and the pigs lifted off into the sky and everybody was happy.

Because he's such a nice guy ("Maybe I'm doing something wrong?) and because I should have paid more attention when he asked that, I'm not even sending him a bill for any of it.

Pigs: to your stations! Fly, you pink porkers, fly!

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

The cure for everything - chmod 777

Linux,MacOSX 2009/10/28

I admit that I have done a "chmod 777" when I should not have. Almost always that came from haste or frustration. Not frustration with Unix permissions, but frustration with whoever had daily care responsibility for the system - their inability to understand permissions might have driven me to this.

Sometimes it hardly matters. At many small businesses, everybody has "got root" anyway and has learned that this magic incantation will "fix" problems. Well, until it breaks a setuid program, of course. Nobody, NOBODY ever learns "chmod +w", do they?

Another rare breakage is /tmp. It's supposed to have the "t" bit set so that only the owners of files can delete, but I've had folks "777" it. Why? Who knows?

More usually the open permissions are applied to some common set of data. All goes well until someone removes (or just moves!) something that is needed by someone else, and then the crying starts.

The most horribly wrong things that can be done with permissions come from people who have learned about "-R" (recursive) or wild cards. Two or three times a year I find a system where someone has done a chmod to ".*". That's bad enough by itself, but when combined with -R, the results can be spectacular.

Unnecessary damage

What you should understand is that this is all unnecessary. Most of us who have to deal with the results of misunderstood chmod's wish that users didn't even know that the numeric form exists. There would be far fewer errors if users only knew the symbolic modes.

The symbolic form is also much more powerful. Consider this :

$ ls -l
total 0
-rw-r--r--  1 apl  apl  0 Oct 27 15:06 a
-rw-r--r--  1 apl  apl  0 Oct 27 15:06 b
-rwxr--r--  1 apl  apl  0 Oct 27 15:10 c

$ chmod a+X  a b c
$ ls -l
total 0
-rw-r--r--  1 apl  apl  0 Oct 27 15:06 a
-rw-r--r--  1 apl  apl  0 Oct 27 15:06 b
-rwxr-xr-x  1 apl  apl  0 Oct 27 15:06 c

ONLY the file that was already executable had full execution bits added - try THAT with Windows!

(If you did want to change all the files, you'd use "chmod a+x a b c")

But I'm being silly. People will continue to "chmod 777" anytime they have a problem. Program doesn't work? Chmod 777. Unexpected error? Chmod 777. Grinding noise inside the computer? Chmod 777. Too hot in here? Chmod 777.

Feeling frustrated by people changing permissions for no reason? By now you should know the cure. Say it with me: Chmod 777.

Don't you feel better now?

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Prevent deletion or moving of files

Linux,MacOSX,Shell 2009/10/27

You need to let users create files in a common directory, but you don't want them to be able to delete other's files. Or you've put certain files, directories or symlinks into a user's home directory and don't want them to be able to mess with any of those. What can you do?

"t" bit

If you create /foo and do "chmod 1777 /foo", you'll have a world-writeable directory with the "text bit" set. Any user can create files here, but they can only delete files that they own (root can still rm anything). That's ownership as listed in the "owner" column of an "ls -l". Group ownership doesn't come into play here although it does change responses a bit.

Let's see what happens when Sam tries to remove Pete's files in a directory with the text bit set:

[sam@localhost foo]$ ls -ld .
drwxrwxrwt 2 root root 4096 Sep 18 06:00 .
[sam@localhost foo]$ ls -l
total 12
-rw-rw-r-- 1 pete pete  29 Sep 18 05:52 pete
-rw-rw-r-- 1 pete apl   29 Sep 18 06:00 peteapl
-rw-rw-r-- 1 pete wheel 29 Sep 18 06:00 petewheel
[sam@localhost foo]$ id
uid=502(sam) gid=502(sam) groups=502(sam)
[sam@localhost foo]$ rm *
rm: remove write-protected regular file `pete'? y
rm: cannot remove `pete': Operation not permitted
rm: remove write-protected regular file `peteapl'? y
rm: cannot remove `peteapl': Operation not permitted
rm: remove write-protected regular file `petewheel'? y
rm: cannot remove `petewheel': Operation not permitted

Now watch what happens when a user in the "wheel" group does the same thing:

[apl@localhost ~]$ cd /foo
[apl@localhost foo]$ ls -l
total 12
-rw-rw-r-- 1 pete pete  29 Sep 18 05:52 pete
-rw-rw-r-- 1 pete apl   29 Sep 18 06:00 peteapl
-rw-rw-r-- 1 pete wheel 29 Sep 18 06:00 petewheel
[apl@localhost foo]$ id
uid=500(apl) gid=500(apl) groups=10(wheel),500(apl)
[apl@localhost foo]$ rm *
rm: remove write-protected regular file `pete'? y
rm: cannot remove `pete': Operation not permitted
rm: cannot remove `peteapl': Operation not permitted
rm: cannot remove `petewheel': Operation not permitted
[apl@localhost foo]$ 
[apl@localhost foo]$ 
[apl@localhost foo]$ rm peteapl
rm: cannot remove `peteapl': Operation not permitted
[apl@localhost foo]$ rm petewheel
rm: cannot remove `petewheel': Operation not permitted
[apl@localhost foo]$ rm pete
rm: remove write-protected regular file `pete'? y
rm: cannot remove `pete': Operation not permitted

Having write permission makes rm proceed without caution, only to be brought up short by the restrictions of the "t" bit.

mount --bind

If the problem is removal of a directory and it is not terribly inconvenient for you to have that directory actually be on a separate filesystem, then "mount" can make the directory safe from removal. You can read more at mount --bind, but it's not very complicated. Let's say we have /dev/foo mounted at /foo and I want a "link" to that under /home/fred. All I have to do is:

mount --bind /foo /home/fred/foo

Fred can have full write permissions on /foo if he needs it, but he will not be able to remove /home/fred/foo. Not even root can:

# rm -rf /home/fred/foo rm: cannot remove directory '/home/fred/foo': Device or resource busy

Now THAT is removal protection!

ACL's

Typically, ACL's let you avoid complicated groups by setting specific permissions for specific users. Other than setting a file as "immutable" (chattr +i filename on some Linuxes), you really can't prevent removal of a file. Of course setting it that way may also make it useless, as even the owner can't modify or remove it either without doing "chattr -i" first.. See ACL's for more on that.

[pete@localhost foo]$ id
uid=501(pete) gid=501(pete) groups=501(pete)
[pete@localhost foo]$ chattr +i pete
[pete@localhost foo]$ rm -f pete
rm: cannot remove `pete': Operation not permitted
[pete@localhost foo]$ mv pete /tmp/
mv: cannot move `pete' to `/tmp/pete': Operation not permitted
[pete@localhost foo]$

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Plenty of space here

Shell,Linux,MacOSX 2009/10/21

"Type ess-see-pee-SPACE-john-ATSIGN-192-DOT". The person at the other end of the phone line interrupted me: "Hold on, I typed an extra space".

"You mean before 'john''?", I asked, "That doesn't matter".

"Why doesn't it matter? It always has", he asked, obviously confused.

I could understand that. Like many people, my customer doesn't understand spaces in command lines. Unlike most, he's not one to leave spaces out - no, he's more likely to insert gratuitous spaces where none are wanted. For example, in an earlier "scp" command, he had added a space after "john". I had chastened him not to add any spaces unless I specifically said to, so it seemed reasonable to him to think that two spaces in a row would also be an error. I needed to explain that multiple spaces would be ignored, indeed not even seen by "scp".

His confusion is easier to understand than that of people who leave spaces out. I understand that part of that came from DOS. Being able to do things like "DIR/P" created a lot of this. But can that really be responsible for all of it? Many people today might have never, ever used a DOS command line - why would spaces still confuse them?

Putting a space after "john" isn't illogical. The "scp" command could have been written to parse the user name as a separate argument. You'd probably need "to" and "from" to specify the direction of the copy; you'd type things like "scp /tmp/foo from 192.168.7.2 john /tmp/foobar" and change "from" to "to" if you needed to go the other way. Some people might find that less confusing than the present syntax.

But if our goal is to not confuse users, maybe we shouldn't have command line arguments at all? Our user would type "scp", press enter, and the program itself would ask the appropriate questions. For a command like "scp", which is meaningless without additional parameters, it wouldn't be difficult to design it to act on any supplied arguments and prompt if none were offered. Interfaces like that are hardly without precedent, but few Unix/Linux commands bother to do that, simply because it's not so easy to do. Take a refresher look at "man scp" and think about how many questions you'd need to ask!

Spaces will continue to confuse for as long as there are command lines. I'll be intoning "ess-see-pee-SPACE-john-ATSIGN-192-DOT" for at least a few more years.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Follow the bouncing dependencies by Bill Mohrhardt

Linux 2009/10/08 by Bill Mohrhardt

I am working on building a Linux box to replace our old SCO Openserver machine, and I had a spare Digi host card and concentrator, so I needed to install & configure those. Red Hat Enterprise Linux 5 was not listed, so I called Tech Support to ask which version I should try. The tech said to try dgap-1.3-15.src.rpm, the most recent beta version of dgap from Digi's website. He said that if I wanted support I would have to pay $75, as the PCI host card whose serial # I gave to him had its warranty expire in 2004. Fine, I proceeded on my own, as usual.

Now, given that I installed RHEL 5.3 from rpms, and did not compile my own kernel, I figured that I would be missing some packages, but it turned into a major pain. First of all, Digi's instructions require rpmbuild, so thanks to pbone.net, I got rpm-build-4.4.2.3-9.el5.i386.rpm, and that installed nicely. Then, the dgap source said that it needed elfutils, so I eventually found the elfutils matching my version on RHEL5 elfutils-0.137-3.el5.i386.rpm. Oddly, when I checked rpm -qa | grep elfutils, my RHEL5 had elfutils-libelf-0.137-3.el5 and elfutils-libs-0.137-3.el5, but not just the plain elfutils.... Whatever, that rpm'ed fine. But then, the dgap install got most of the way finished, and complained that it needed curses (funny, I had already been providing some of those!). When I checked my rpms, I had ncurses but not curses. I tried setting curses as a symlink to the most recent ncurses library but that did not work. I Googled around, and one linux site said that I needed the ncurses-devel package, but when I downloaded and tried to rpm that one, it said that it needed some other libraries, so I decided to go home at that point.

The next day, I Googled around some more, and one site said that when downloading ncurses-devel, you must make sure to get the same version as your ncurses. My RHEL5 had ncurses-5.5-24.20060715, and again pbone.net had the matching rpm for that, so I downloaded and rpm'ed, and that went through no problem. With the ncurses-devel library in place, the rpmbuild --rebuild dgap-1.3-15.src.rpm actually worked! So now I had the dgap rpm and again the rpm command installed that. Now, I try mpi to configure the card, and the menus get to the point where I choose the type of card (Digi C/X), and then seems to hang. Now I did a ps -ef | grep pts/1 to see if it was a process gone awry, but there was no CPU usage on that port, and there was a long command listed as a process that appeared to be the next question that the mpi menu was trying to ask. After some muttering, I figured that it must be trying to do some graphics that would not display on the telnet session, so I killed it and executed it from the console. That did the trick. I then did the chkconfig --add dgap to add it to the boot-up list.

Then I was able to connect an old Wyse 160 monochrome terminal (yes, I still have one), and I set the baud rate with ditty ttya12 speed 57600. Then I sent echo 'Hello' > /dev/ttya12, and the polite greeting appeared on the terminal. Then I added a12:2345:respawn/sbin/agetty -L ttya12 57600 wy60 to /etc/inittab. To get it to re-execute with the changes, I then did telinit q. A login prompt appeared on the wyse terminal, and I started testing the application programs.

We still use the PCI-card-based Digis, as we have 5 active C/CON 16 concentrators at one location (linked via several pairs of Digi Fiber Modules), and another 2 at our other site. The nice thing about dumb terminals is that they just work, with no maintenance, and nobody messing around on the Internet. We don't use the monochrome Wyse units anymore, we use Spotline terminals that take any colour monitor and keyboard.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Is Linux the One True Religion?

Linux,MacOSX,Lighter 2009/10/06

I don't know when or why I wrote this. It couldn't have been too long ago; I first bought an iBook in December of 2002. It apparently was in response to someone who said their professor advised them not to use Linux. Whatever, whenever, here it is:

Oh, I found it at Plex86.org. I thank them for preserving it - I had forgotten it entirely.

There is only one true religion, though it's impossible to know whether it's Linuxism, BSDism or SysVism. Some extremely deluded folks once thought it was McBridism but that's generally considered to have been falsified. So you have a one in three shot at salvation.

The problem is, if you guess wrong, you are doomed to spend eternity crawling through the Mines of Misery with a Tandy Model 100 computer as your only companion. The Gods of *ix are jealous gods and won't stand for competition and false idolatry.

The only certain thing is that the Microsoft guys are screwed. If you worship at the altar of Linux, you may end up pecking at the Model 100's keys while the stench of Unicorn dung fills your nostrils, but at least you have a shot at your own Camel Beast and all the other perks that go along with having chosen the Right Religion.

My feeling is to embrace 'em all and believe in nothing. You might call me a Secular Posixist. I'm typing at an iBook keyboard, have one terminal window open to a Linux website I run, and another to my BSD based site. May the gods have mercy on my soul, but I still have some SCO customers here and there (though I've moved more than a few to Linux).

Who cares what some P-thing (Pastor or Professor) thinks? Linux is not real Unix, but Unix isn't real Linux either. Think for yourself - it may seem strange at first, but it's habit forming. Just imagine making up your own mind about something.. I know, it's scary taking responsibility for your own opinions. When you do that, you can't say "God says I have to hate gays" or "My professor says I shouldn't use Linux". Nope, you'll have to take full responsibility for your own opinions. I don't mean that you have responsibility to other people necessarily; primarily your responsibility is to yourself. You are the captain of your ship and shouldn't be letting other people tell you how to trim the sails. Sure, you'll listen to the tales of those with more experience, but remember that ultimately it's your course to chart, not theirs.

So give the Professor a big old smile and agree wholeheartedly:

Yessiree, Linux is not real Unix. And so?

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here

Network Know-How

Books,Reviews,Networking,Microsoft,Linux,MacOSX 2009/10/07

Index by Subject

Network Know-How
John Ross
1593271913

This is the book I'd give to someone who needed to learn a lot about computer networking quickly.

I'm tempted to say that this isn't a geekish book, but it is. It just doesn't read like a geekish book. You'd need to be very tech-phobic to feel frightened by this: the author explains things very gently, yet very completely.

I was very impressed that he did not ignore Linux or Mac OS X. The text regularly refers to those other Operating System choices and he has a few pages that give a very fair comparison of their respective stregths and weaknesses. Not only that, but he recognizes that:

In a very small business, the numbers will probably work out in favor of free or inexpensive server software unless you have to pay for outside support.

Oh, I could quibble that he didn't need the "outside support" qualifier and didn't need to restrict this to "very small business", but as most advice of this sort is often very Windows biased, I was happy to see that.

This is complete, from running wiring to what you can do with a network after it's working. It covers wireless security, vpn's, print servers, setting up routers, everything. What made it particularly great for me was that as I read along, I'd start to think, "Ah, but you are ignoring.." and then, bam, he'd hit my complaint in the next paragraph.

Very, very well done. I have to give it a top rating.

Tony Lawrence 2009-10-06 Rating: 5.0

Order (or just read more about) Network Know-How from Amazon.com. Yes, I make a small referral fee if you use that link.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

Skills Tests

Psst - wanna work for yourself?

Unix/Linux Troubleshooting e-book

Kerio Mail Server

Consulting

Advertise Here