Security Site News at A.P.Lawrence.com

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. If you have any question, please do feel free to contact me.

I don't WANT the Internet to forget!

Web-HTML,Blogging,Security,Opinion 2009/10/23

I was listening to an NPR show about internet privacy and the "worrysome" fact that internet information lasts forever. The very forgettable guest being interviewed was harping on "forgetting" - he apparently wants us to be able to set retention dates for things the Internet knows about us. Callers chimed in with stories of real and potential embarrassment from discretions and more serious actions that their boss, their children or their spouses might accidentally discover while bumbling about the Internet. SOMETHING MUST BE DONE!

I don't agree.

If you want to go digging around, you can find some "embarrassing" stuff about me on the Internet. That is, you'd find stuff that you might THINK would embarrass me and probably would embarrass whoever that "let's forget it" guy is and apparently could upset some of the people who called in all worried about something they said or did in 1994. As for me, I don't care. If you aren't smart enough to realize that EVERYBODY has skeletons in their closet, that EVERYBODY has been petty, vain, jealous, stupid, dishonest, and worse, why would I care what you think about me?

Wouldn't we all be better off if we stopped pretending that we are perfect or even close to it? I'm not saying we shouldn't strive toward not being jackasses, not doing dumb things. I'm saying we should accept that we are human, we do screw up and we and everyone else just need to get over it.

Maybe if everyone's "dirt" was always easily dredged up we could dispense with this fantasy of saintly people passing through their oh-so-perfect lives without any stain of error. Maybe if nobody could hide their indiscretions and mistakes, our children would better know how to avoid or mitigate their own?

Say it with me now: I can be a jackass and so can everyone else. I have done stupid things, cruel things, idiotic things and so has everyone else. Anyone who presents a perfect facade to the world has dirt behind the curtain and is lying to us overtly or by omission.

If I'm trying to find out what kind of person you are and I come up with nothing, what have I learned? What are the possibilities? Either you've been really sneaky and secretive or you are such a timid, inactive and uninvolved person that you've never had an opportunity to screw up. Do I really like either of those? No.

Let's stop being phony. People screw up. Maybe there are a few untainted people somewhere, but most of us wouldn't like them because they probably have no fire, no spark, nothing to make them interesting. They walk through life so carefully, so fearful of error - what clods!

So no, I don't want the internet to forget anything about me. I want it all preserved forever. I want my future relatives to be able to learn things about me that I can't learn about my ancestors. I want future historians to have a treasure trove of data that will tell them societal secrets that are almost never known about past generations. I don't WANT the Internet to forget!

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

A fish is not a hack

Security 2009/10/07

Just before we turned in last night, I heard a scary teaser on the news: an announcer breathlessly asked "Has your email account been hacked? Do you use Gmail or Hotmail? Your information may be at risk!"

Oh damn. I went to my computer and typed in "Gmail hacked" and sure enough, found a news story that said the same thing. Arrgh - I immediately changed my and my wife's passwords.

This morning, after reading better news sites, I found that nothing was really "hacked" - this was just yet another phishing scheme that caught a few thousand people stupid enough to fall for it.

Well, OK: it's been a while since we changed those passwords anyway, so that's fine. However, it's annoying that news media doesn't distinguish between a true hack like someone breaking into Google and a phishing exploit like this. That newscaster should have asked "Are you a gullible fool who will give out their password when asked? Too bad for you, more at 11:00."

Of course, phishing exploits can include a real security breach component. If someone poisons DNS so that your attempt to access Gmail brings up a login page that looks like Gmail but is not, I could agree that was hacking. But if you are dumb enough to fall for an email that asks you to click on a bogus link, well, the problem is with you and your lack of common sense.

Apparently this was only about 10,000 accounts. That's actually pretty good if the bait was wide spread - if only that many fell for it, there might be hope that the general public is getting smarter about this stuff.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

How to prepare and plan for Incident Response by Michael Desrosiers

Security,MDesrosiers 2009/10/02 by Michael Desrosiers

This post has been removed at the request of the author.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

SpamCheetah
Stop spam dead in its tracks!

Powerful crypto from the UNIX command line by Girish Venkatachalam

Security,Programming,Girish 2009/09/20
by Girish Venkatachalam

Girish Venkatachalam is a UNIX hacker with more than a decade of networking and crypto programming experience. His hobbies include yoga,cycling, cooking and he runs his own business. Details here:

http://gayatri-hitech.com
http://spam-cheetah.com

I will be really surprised if a linux distro ships without OpenSSL pre installed. I know that it comes by default since it uses a very liberal license. The importance of OpenSSL toolkit for crypto cannot be overestimated.

OpenSSH project relies on OpenSSL for its backend crypto operations. Though named as an SSL library, it is a very comprehensive toolkit with all kinds of facilities related to cryptography. Even simple things like random number generation, base64 conversion, file integrity checking with SHA1 or prime number generation can be done with the OpenSSL command line tool.

OpenSSL is a crypto toolkit and it produces two libraries libcrypto.so and libssl.so. You could also link to it statically. The main purpose of the OpenSSL project is to provide library facilities for various operations related to SSL. But over time it has evolved into a fantastic command line utility that gets our job done as long as we know a thing or two about crypto.

Knowing a thing or two about crypto is not so easy. Most people have no idea what is meant by PKI. Most people do not understand why RSA keys are 1024 bits and why AES keys are 256 bits. How can the AES key be stronger? Crypto theory is very deep and mathematical in nature. And OpenSSL programming requires very advanced C skills. I remember how I struggled when I worked with the guts of OpenSSL. But for this article, we only want to look at some of the really useful features offered by such a rich toolkit. All from the command line.

OpenSSL also has a shell interface which gets invoked when you type openssl like this:

$ openssl                                                                      
OpenSSL>

But we do not have to go inside the shell at all. We can use command line switches for most of our needs. Nowadays you also have GnuPG for performing command line operations like file encryption and message signing. OpenSSL also comes in very handy when you wish to generate server certificates for your Apache HTTPS service. Or you might wish to create a public keypair for accessing your IMAP over SSL account or SMTP over SSL e-mail account.

Let us take a few simple examples.

File integrity checks help us detect file corruption and change. We want to detect a truncated upload or download. We also want to know whether a binary file has changed. With text files you can always use diff. Diff also is useful with binary data but the best approach would be to use a strong fingerprinting algorithm like SHA1 checksum.

There is a command called sha1sum on most linux boxes similar to the cksum utility of yore. OpenSSL can also do the same thing with this command.

$ openssl sha1 /etc/passwd                                                     
SHA1(/etc/passwd)= 61293afc53dd8465a28d49fc1d11676badcd0076

A SHA1 digest output has a constant length of 160 bits and it is in a binary format. This output has 40 bytes because the binary output is Hex encoded. BCD representation takes one byte of binary data and represents it as two ASCII bytes. Hence the output here is 40 characters/bytes.

Let us generate some random data just for fun.


$ openssl rand 1024

Oops. There is a problem here. Your screen gets garbled. What to do? We do not want raw binary data spat to our beautiful terminal. Instead we want to do one of 3 things. We could get a Hex encoded output like above, we could base64 encode it or we could write it to a file by redirection or ask OpenSSL itself to do it with the -out switch.

Base64 encoding is different from the BCD hex representation we saw above. OpenSSL is good with ASN1 encoding and Base64 encoding in addition to crypto stuff. Base64 uses 3 bytes of binary data and create 4 ASCII characters as output. So there is a fixed 33% increase in filesize when you base64 encode a file. Of course you have padding and other variations, but this is the simple math. 3 x 8 = 4 x 6 = 24.

Base64 output uses 2^6 or 64 ASCII characters to represent a byte. Nowadays Base64 encoding is used everywhere. A good example is for e-mail authentication protocols. SMTP authentication uses DIGEST authentication using base64 encoding of passwords.

Evidently OpenSSL is a lot more than this. I hope this gets you started.

References

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

SELinux:; Tech Words of the Day

,Linux,Security,Misc.

SELinux - love it or leave it

September 2009

SELinux is a Mandatory Access Control system. You can get an idea of how you'd configure this at my Selinux on FC5 article even though it's a few years old, but before you read that, just let me put on my flame proof suit here... ok, I'm ready: I almost always disable SELinux sooner or later.

Yeah, really. I think SELinux is a wonderful, wonderful idea. If used properly, it really can help make your system more secure. My problem is that it doesn't get used properly. So upgrades get done and the people who do the scripts forget all about SELinux and what YOU get is a broken app or a broken system. This happens all too frequently, and unless you are willing to have constant vigilance, SELinux (or lack of due digilence with regard to it), will bite you sooner or later.

So - if you are willing and able to babysit this and be sure everything has been properly reset after updates, yes, use SELinux. If you are like my more typical clients, you may not want to invest the time.

When I do an install, I usually leave SELinux enabled. The first time I get the "everything is broken" call because of SELinux, I have a long conversation about all this and the result usually is to disable it because they don't have time to watch this and aren't willing to pay my time to do it for them.

SELinux: great idea. Not so great in the real world. Not SELinux's fault, just the way things are.

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

SpamCheetah
Stop spam dead in its tracks!

Remote OS fingerprinting by Girish Venkatachalam

Security 2009/08/31

Girish Venkatachalam

What is meant by OS fingerprinting?

It must be familiar to UNIX geeks. There are popular tools like nmap that help you identify which hosts run Windows and which hosts run Linux. This can be as specific as even getting to know if a patch or service pack in Windows was installed.

But there is a problem with nmap OS fingerprinting as it uses active fingerprinting. Not a great idea. We want to use passive OS fingerprinting. In passive OS fingerprinting we rely on TCP SYN packets from the remote host to identify the OS. This is quite reliable though it can be trivially spoofed. I would imagine that if we use passive OS fingerprinting we can be reasonably sure about the remote OS.

It can be used as a policy tool to implement firewalling that can protect us against Windows worms or viruses. We can have a logical separation between Windows hosts and other hosts.

Passive OS fingerprinting can help us in many other ways too. We can find out many things that are hidden from the eyes of systems administrators. A tool called p0f is famous for doing passive OS fingerprinting correctly. And OpenBSD pf, the firewall in OpenBSD has inbuilt ability to do fingerprinting. You can also change the string that it displays for identifying the OS by specifying it in a file /etc/pf.os on any OpenBSD machine.

p0f and OpenBSD pf both use the TCP default Window size, time to live, the presence of absence of the DF(dont fragment) bit in IP header, the size of the SYN packet and the options in TCP header to identify the remote OS through passive fingerprinting.

You can identify what software people have installed by looking at the greeting message of TCP protocols by simply connecting to them with netcat. You can know if people use sendmail, postfix or MS Exchange. You can identify the OpenSSH version, you can know which web server people use and many other networking forensic data can be collected.

If you wish to know the countries that hit your web server, then GeoIP can help you lookup IP address and know where the ISP is located. This is not accurate as most free tools don't have the correct database. You have to do some crosschecks before arriving at the right tool.

Network forensic analysis is towing the thin line between hacking and cracking. We are not interested in prying into other people's or other network's innards. But you can use such tools for several useful applications without intruding into other's privacy.

Network scanning is also useful to know which services are running on UNIX hosts and request users to turn off harmful services. NAT is a blessing in disguise because most machines are not accessible to the big bad Internet. If that were not the case we would be having a lot more attacks than now.

References

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.

SpamCheetah
Stop spam dead in its tracks!

How relevant is a good antispam solution for you? by Girish Venkatachalam

Security,Mail,Spam,Girish 2009/08/24

Girish Venkatachalam

How relevant is a good antispam solution for you?

Unsolicited bulk e-mail(UBE) or unsolicited commercial e-mail(UCE) is what is commonly known as spam. Spam nowadays has become such a common "word" that we use it with wikis, with online mail ID creation sites and so on. CAPTCHA is geared towards working around robots and programs that masquerade as humans.

We hackers sometimes turn towards evil ways and most spammer botnets are created by very intelligent but highly immoral crackers who get paid and sometimes paid too well for the work they do. This causes relentless misery for you and me. In fact the problem is so far reaching that anyone with even the slightest exposure to Internet know that p0rn mails are a problem to deal with.

I really wonder how many people can be so naive as to expect unreasonable sexual satisfaction or sudden billions in their bank account. Einstein has once said that the universe and human stupidity are both infinite and that he was not sure about the former.

Human stupidity makes people open such spammy mails that cause further problems for their employers who are eminent baits for phishing attacks and other social engineering attempts wrought by spammer networks.

People have a tendency to expect miracles. Just look at how many money astrologers, sooth sayers and magic healers make. Evidently spam is big business. Or else we will not be talking about spam. And fools continue to fall for them. The world cannot be freed from them. So the only option left with us is to protect them.

How?

Comparison of various approaches

Preventing unwanted e-mails from getting into your Inbox folder is clearly not a simple task. The smartest minds of our times have applied themselves to this problem and people like Paul Graham and Vipul Ved Prakash have helped lesser mortals like us live relatively peacefully. It was for Vipul's razor that Vipul was awarded the MIT Young Innovator award in 2003. That should tell you how important the spam problem is.

Paul Graham is famous for his essays on spam and Bayesian filtering and we find the best implementation of his strategy in spamassassin spam filter written in perl(gulp). What I say here may hurt many of you but I will still say it. Both Vipul's razor and spamassassin are written in perl but spamassassin sucks. And it sucks real bad. Perl is not the language for doing content scanning at wire speed. It can be used for prototyping software and doing non real time work. Well, Apache modules are written in perl but that is a different story.

I don't have a problem with spamassassin just because it is written in perl. I don't like it because it does a very nasty job of spam control. It is complex, slow and causes false positives, quarantines and what not. Unfortunately it seems hugely popular. Well well.

There are many approaches taken to save vulnerable people from clicking at nasty spammer ads. I have never fallen for domains like paypal.us or bankofamerica.foobar.com or whatever. But there are many who do. The best thing to do would be to not allow such mails to attract their attention. How?

By doing it at any cost. By any cost is meant that even if you lose legitimate mails, we cannot allow dangerous mails in. This is a measure of desperation taken by companies that are left with no choice.

And open source solutions like spamassassin make the problem worse by making people believe that you cannot make omelettes without breaking eggs. If you want no spam, then you also might lose important mail. And people don't buy products based on technical merit.

People buy products based on brand name. People buy products depending on what is the coolest thing in town. People buy what other people buy. They discuss with friends and like minded people and then decide. They also want to escape responsibility and consequently they do not wish to risk their reputation. So even if you give them nectar, they will continue to use existing poison because they know its taste.

Known devil is better than unknown angel. But there are companies where decisions are taken by people other than systems administrators and half baked technicians. And companies exist which don't need to answer somebody else about what they choose. My customer was one such. He is the proprietor of the company and he knew a thing or two about open source. And he somehow felt that I could be trusted.

This level of personal interaction cannot be replaced by the excellence found in the open source world. It takes time for adoption.

But I firmly believe that ultimately if your product is good you will win. Nature and the law of karma works inexorably and with astonishing accuracy. The tough and the capable survive. The rest are left in the lurch.

Now let us get back to the technical problems associated with spam control. Spam comes in various shades and colors. It is impossible to accurately define what is spam. There is a comment often made that what is spam for one may be ham for another. This is bullshit. Before google came, Altavista never thought that web search should be done the way google does. And you know the rest.

People clearly know what spam is.

Even if you are really interested in knowing about products that enhance your private organs, if you don't ask for a mail, or subscribe to a newsletter and if you receive it, it is clearly spam. As simple as that.

There is one definition of spam that I like a lot. Spam is bulk mail sent by Botnets(automated programs that send out mail) to millions of unrelated recipients. They pump traffic at such high rates that most of the IP overloads in the Internet are caused by such criminals.

So according to me, spam is nothing but Botnet spew. I like this term a lot. In other words, spam is what is generated by computers and sent by computers. If humans send mail, even if a mailing list is used or even if it is addressed to 1000s of recipients, it is not spam. It is solicited and coming from a human. You may think of it as spam but it is impossible to make a machine make a decision or an algorithm come to a conclusion. Consequently we have to conclude that spam is not unwanted mail. Spam is bulk mail or commercial mail.

The mathematics of the spam problem

Spam control math is no great shakes. For that matter even google's search algorithms are easy to understand. It is only the details that need genius to understand and troubleshoot. The basics of math are always easily understood by common sense.

I learnt Bayesian probability and statistics as part of my engineering degree long ago. It is a very simple concept. If you throw a fair die two times, the probability of getting 6 both times is a product of the probability of getting 6 the first time and the probability of getting 6 the second time.

Physical independence of probabilities is known by Bayesian probability theory.

See? It is not so complicated after all. If you read Paul Graham , you will find that he has expressed the same idea and how it relates to spam in many words. He is a great writer no doubt and his idea of applying this simple mathematical truth to a practical and relevant problem like spam is highly commendable. But read the next paragraph.

But the assumptions made by Paul Graham are not sound. You can have spammy content in legitimate mais and vice versa. So no matter how brilliant or adaptive or well performing your algorithm is, its delicate clockwork will blow to smithereens when fed with unexpected data.

Perhaps I should tell you how Bayesian theory relates to spam control. Two spammy words appearing close to one another have a certain probability of occurrence in spam. And a different probability of occurrence in legitimate mail. This can be compared to the two probabilities of a fair die giving a result of 6 both times.

CRM114 discriminator uses another simple mathematical concept called Markovian chains to further refine this algorithm. Whereas Bayesian probability can only account for characters and words occurring together in headers and mail bodies, Markovian chains have the mathematical ability to construct databases with even sentences. Evidently this is a lot of work and you require very powerful processor, memory and of course disk space.

And people only talk about spam efficiency. We stop 99.9999% of the spam messages. Now is this calculated as a percentage of total mail received(including spam) or is it taken against the spam messages that you did not get or is it some other metric coming from an obscure database? God alone knows.

Moreover they conveniently ignore the false positives problem. Your spam filter is great, it stops 99.999% of the spam. But you lost an important mail for an appointment with your boss. How good is your filter? Products usually do not mention this. They will provide you with technical support and spend time with you but what about the lost e-mail?

To mitigate this, many vendors have a concept called the 'quarantine'. It is a wholly unnecessary overhead invented for business purposes alone. It serves no practical purpose. Commercial interest forces people to show how much work they are doing. And sometimes the burden falls on your head. You should maintain the product and babysit it and manually interfere. You should pass 'parked mail messages'.

My customer had a harrowing time even after buying my product because I was present in the server room when his sys admin would painfully delete the mails and pass mails in the quarantine for the other domain for which he had not purchased my product. He had spent a lot of money buying the product. He could not throw it into the drain after all? Could he?

The other math involved in spam control is the math of the rsync algorithm. Or the checksum computation involved in cryptographic signatures. If you know the MD5 or the SHA1 message digest algorithm, you know what I mean. It is vaguely similar to symmetric encryption as it also involves multiple rounds of similar operations in math(mainly EX-OR and matrix multiplication), but you get a constant value as result. With a unique mathematical property that no two inputs can give the same output. MD5 gives a constant value of 128 bits as output and SHA1 gives 160 bits as output. Even the slightest change in input will cause widely varying output.

rsync uses this algorithm to detect file changes and a rolling checksum is computed after splitting files into blocks. And Vipul's razor and DCC use the same concept with a twist to detect spammers modifying their messages to send to other innocent bystanders on the Internet.

The careful reader will notice that this approach has problems too. I think that gmail uses this approach. We need a corpus of spam and this approach necessitates manual intervention. A global database of spammy content is required to feed the checksum computation engine and that is what is used to prevent others from getting the spam with modifications.

This approach naturally takes us to the next section. So now we are striving towards a better understanding of spammer mentality and motives. We are coming closer to the real world and consequently we can avoid the idealistic assumptions that Paul Graham's Bayesian approach had.

The business model of spammers

SpamCheetah and OpenBSD greylisting based products/approaches understand the psychology of spammers and appreciate the practical side of spam propagation and spam generation. Spam is not something that comes out of ether and escapes into the void. It is generated by humans that set machines into motion , and they can masquerade IP addresses, they can masquerade sender e-mail addresses, they can fake several other things and generate bounce messages or do backscatter, but there are certain things that they need to strictly abide by if they want to deliver their spam.

This fact will never change. Spammers want their message to land in your mailbox. And they want you to open in. It may be in Korean or Chinese, it may be image spam, it may be something else, but they cannot get around this basic fact.

The second truth that one has to realize is that spammers do this for money. They don't do this for fun and they don't do this for attaining nirvana. They do this because marketing sells. And e-mails cost nothing. They have to pay service providers and the websites the spam mails point to help out with them.

They also face the same problems that criminals face in every country. They have to grapple with the legal system. They have to face opposition, complaints and sometimes even punishment. So what do they do? They start operating in a clandestine fashion and operate in a manner that helps them escape detection. Hence Bogons which are unallocated IP addresses of BGP prefixes that spammers use to pump traffic, and once the harm is done, they go to some other location and wreak havoc. You have seen movies in which the villain has multiple identities and passports and how they fly to other countries.

We don't get sufficient time to react. As I said before, most of the traffic overloads in Internet routers are due to worms and spam. And if we knew their source, we could always plug the holes. But this is easier said than done. But we can protect ourselves with the right medicine.

What is the right medicine?

You can force spammers to pass a test which we keep for both innocent people and criminals. We know the innocent people will pass and that the criminals will get caught. And this is the test performed by greylisting and tarpitting. This is further helped by IP address blacklisting and e-mail address whitelisting. There are idiots amongst spammers who get caught and there are databases who track such current BGP netblocks that are known to send spam. SpamCheetah uses all the 3 approaches viz, greylisting, tarpit and blacklist of known spammers.

But greylisting has a problem. People don't like it because it delays the first mail from a domain that has never contacted you before. In practice this is never a problem but people are people after all and their anxiety needs to be addressed. And people come with baggage. Greylisting is an old concept and until now, nobody implemented this approach correctly. Design is one thing. Implementation yet another.

You need to mix greylisting with some pepper and salt to create the right medicine. And this recipe is reducing the TCP window of the SMTP dialogue. This is done by the OpenBSD tarpit. It takes genius to come up with such an idea but it is a very powerful concept. You not only send back the error message of 403 or 503 to the sender, you also subject the sender to another acid test.

You reply at the rate of 1 character per second. This can be very annoying for someone who wants to deliver million messages but for legitimate human generated senders this is nothing. Yet another application of understanding real life and nature better.

The greatest side effect of the science and math of OpenBSD greylisting is that I can now give spam control in a USB stick. Not the 16 Gig one, but in 1 GB. And you can run it in a box with tiny processing power and memory. After all we don't do mail. We don't need hard disks. We don't need to talk at high speed because our job is to talk slowly, and we don't need high processing power as we don't have to do content scanning. We only need the CPU to run our daemons that track IP addresses in our database. We don't store too much data either. More details here.

The other great side effect or benefit of OpenBSD greylisting is that you don't even allow the spammer to deliver the message to you. So I cannot prove that you would have received spam. I never receive it. I never allow the spammer to consume my bandwidth. Now how can I prove that I achieve x% spam catch rate? It is impossible. I save you precious bandwidth, mailbox storage space, backup costs and free your network for productive activity. Of course I can show that if you don't run this filter, you receive spam. That is all.

This is a great example of technical superiority enabling unimaginable possibilities.

References

Comments: Click Here.

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them.