Tidy Tutorials

Welcome to Tidy Tutorials

noreply@blogger.com (righteous) — Fri, 02 Nov 2012 11:49:00 +0000

Tidy Tutorials is the best place for C++ examples and Java examples. Code snippets and tutorials covering topics such as templates, pointers, dlls, JNI, JNDI, J2EE and JWSDP. Also covered are topics on Apache, XML, Glassfish, MySQL and Oracle. Easy to understand step by step rutorials and demonstrations. Learn C++ and Java Object Oriented principles, idioms and patterns.

Tidy Tutorials also has scripting examples for Php, Perl and Javascript. Use regualar expressions to create web crawlers and proxies in PHP or Perl. This site has tutorials on javascript and AJAX and explains how to use available apis to create rich and interactive widgets.

C++, Java, Javascript, Perl, PHP we have all the examples. Easy to follow, easy to understand step by step tutorials.

Learn C++ and Java. Learn scripting languages such as Perl, PHP and Javascript. Become a guru and master developer. Code Diaries has tutorials and examples to get you there.

CodeDiaries Proxy Network - For free web and ip:port proxies using the Code Diaries proxy core.

Dirt Cheap Proxies - For fast and reliable managed proxies for $5/month

Writing a class to Random Access File via reinterpret_cast

noreply@blogger.com (righteous) — Fri, 28 Sep 2012 05:22:00 +0000

Aim
The aim of this tutorial is to create a template that can be used to write and read classes to and from a random access file. This tutorial can be used as an introduction to templates, reinterpret_casting from classes to char array and how to implement a random access file.

The first file is TRandomAccess.hpp and it contains all the template code. The second file main.c is an example of how to use the templates.

The basic idea is to cast the classes into byte arrays via reinterpret_cast and then write them to a file. Using fseek we then travel to any point of the file to retrieve or write a class.

 1. /*Righteous Ninja AKA P.S. Ching codediaries.com*/
 2. 
 3. #ifndef TRANDOMACCESS_H
 4. #define TRANDOMACCESS_H
 5. 
 6. 
 7. #include <stdio.h>
 8. 
 9. #define    RADOMPOS_APPEND    -1
10. #define RANDOMF_ERROR    int
11. 
12. enum{
13.     RANDOMF_OK=0, RANDOMF_OPENERR, RANDOMF_SEEKERR, RANDOMF_EOFERR, RANDOMF_BADPOINTER
14. };
15. 
16. template < class T>
17. class TRandomAccess{
18. public:
19.     TRandomAccess();
20.     ~TRandomAccess();
21. 
22. public:
23.     RANDOMF_ERROR Open(char*);
24.     void Close();
25.     RANDOMF_ERROR Write( T*, int=RADOMPOS_APPEND);
26.     RANDOMF_ERROR Read( T*, int);
27.     RANDOMF_ERROR ReadNext(T*);
28.     void ResetReadNext(int);
29. 
30. protected:
31.     FILE* random_fp;
32.     int   nextPosition;
33. };
34. 
35. template <class T> TRandomAccess<T>::TRandomAccess(){
36. }
37. 
38. template <class T> TRandomAccess<T>::~TRandomAccess(){
39. }
40. 
41. template <class T> RANDOMF_ERROR TRandomAccess<T>::Write(T* in_t, int position){
42.     if(position == RADOMPOS_APPEND){
43.         if( fseek(random_fp, 0, SEEK_END ) != 0)
44.             return RANDOMF_SEEKERR;
45.         char* tmps = reinterpret_cast<char*>(in_t);
46.         fwrite(tmps, sizeof(T), 1, random_fp);
47.     }
48.     else{
49.         if( fseek(random_fp, position*sizeof(T), SEEK_SET ) != 0)
50.             return RANDOMF_SEEKERR;
51.         char* tmps = reinterpret_cast<char*>(in_t);
52.         fwrite(tmps, sizeof(T), 1, random_fp);
53.         fflush(random_fp);
54.     }
55.     return RANDOMF_OK;
56. }
57. 
58. template <class T> RANDOMF_ERROR TRandomAccess<T>::Read(T* in_t, int position){
59.     char tmps[sizeof(T)];
60.     if( fseek(random_fp, position*sizeof(T), SEEK_SET ) != 0)
61.         return RANDOMF_SEEKERR;
62.     if( fread(tmps, sizeof(T),1, random_fp) == 0 )
63.         return RANDOMF_EOFERR;
64.     if(in_t!=0){
65.         memcpy(in_t, reinterpret_cast<T*>(tmps), sizeof(T));
66.         return RANDOMF_OK;
67.     }
68.     else{
69.         return RANDOMF_BADPOINTER;
70.     }
71. }
72. 
73. template <class T> void TRandomAccess<T>::ResetReadNext(int t){
74.     nextPosition=t;
75. }
76. 
77. template <class T> RANDOMF_ERROR TRandomAccess<T>::ReadNext(T* in_t){
78.     return Read(in_t, nextPosition++);
79. }
80. 
81. template <class T> RANDOMF_ERROR TRandomAccess<T>::Open(char* filename){
82.     random_fp = fopen(filename, "r+");
83.     if( random_fp == 0 ){
84.         random_fp = fopen(filename, "w");
85.         if( random_fp == 0 ){
86.             return RANDOMF_OPENERR; 
87.         }
88.         return RANDOMF_OK;
89.     }
90.     else
91.         return RANDOMF_OK;
92. }
93. 
94. template <class T> void TRandomAccess<T>::Close(){
95.     fflush(random_fp);
96.     fclose(random_fp);    
97. }
98. 
99. #endif

Hide line numbers

Below is the main.c, which shows how to use the TRandomAccess template.

 1. #include <time.h>
 2. #include <string.h>
 3. #include <stdio.h>
 4. #include <stdlib.h>
 5. 
 6. #include "TRandomAccess.hpp"
 7. 
 8. class man{
 9. private:
10.     time_t dob;
11.     char name[100];
12. public:
13.     man(char* name){
14.         dob = time(0);
15.         strcpy(this->name, name);
16.     }
17. 
18.     void details(){
19.         printf("%s %d\n", name, dob);
20.     }
21. };
22. 
23. int main( int argc, char** argv){
24.     man * m1 = new man("douglas adams");
25.     man * m2 = new man("captain kirk");
26.     man * m3 = new man("michael schumi");
27. 
28.     TRandomAccess<man> manfile;
29. 
30.     manfile.Open("man_store");
31. 
32.     manfile.Write(m1);
33.     manfile.Write(m2);
34.     manfile.Write(m3);
35. 
36.     manfile.Close();
37.     
38.     manfile.Open("man_store");
39. 
40.     man * manout = (man*)malloc(sizeof(man));
41.     
42.     manfile.Read(manout, 1);
43.     manout->details();
44. 
45.     for(manfile.ResetReadNext(0);manfile.ReadNext(manout)==0;){
46.         manout->details();
47.     }
48. 
49.     man * m2r = new man("james T kirk");
50.     manfile.Write(m2r, 1);
51. 
52.     for(manfile.ResetReadNext(0);manfile.ReadNext(manout)==0;){
53.         manout->details();
54.     }
55. 
56.     manfile.Close();
57.     free(m1);free(m2);free(m3);free(m2r);
58. }

Hide line numbers

Linux C++ Overlapped Server and Client Socket Example

noreply@blogger.com (righteous) — Thu, 28 Jun 2012 11:38:00 +0000

Aim
The goal of this example is to create simple Linux C++ server that accepts socket connections and uses overlapped IO - ayncrhonous non blocking sockets - to service multiple sockets while being single threaded. This covers all the steps required to open a C++ client/server socket, set it to non-blocking and the basics of using the POSIX asynchronous API such as using 'select' and FD_SET to test the descriptors. Please leave any comments or questions at the end of this tutorial and I will endeavour to answer them. This tutorial shares the client code with Linux C++ Socket Example with Client Server and Mulit-Threading .

Assumptions
This article assumes that you have a Linux based C++ compiler installed and configured. You can use cygwin if you do not have a Linux environment. Also, this tutorial assumes that you have familiarized yourself with the Linux C++ forked and multithreaded examples available at this site. Overlapped io can be a little difficult to digest so it's best to be familiar with sockets, compiling and running the demo

Versions used in this example

Sofware/Component	Image
Fedora Core 5	N/A

Links to these files can be found here

You can download the zipped example here.

Steps required for a client socket

Linux C++ Socket Example with Client Server and Mulit-Threading

Write and compile the Server

The server listens on port 1101 for a connection. When it receives a connection it puts the socket into an array of file descriptors. It then uses select to test the file desctiptors to see which are ready to read and which are ready to write. Please be warned that the overlapped model in this example is an extremely simplified model and it doesn't reuse free'd sockets from the list of sockets, so you will ultimately exhaust the list. Rewrite this to use colsed/free'd sockets and you will be fine.

 1. #include <fcntl.h>
 2. #include <string.h>
 3. #include <stdlib.h>
 4. #include <errno.h>
 5. #include <stdio.h>
 6. #include <fcntl.h>
 7. #include <netinet/in.h>
 8. #include <resolv.h>
 9. #include <sys/socket.h>
10. #include <arpa/inet.h>
11. #include <unistd.h>
12. 
13. #define NOOF_SOCKETS 100
14. 
15. int main(int argv, char** argc){
16. 
17.     int host_port= 1101;
18. 
19.     int i;
20. 
21.     int socketlist[NOOF_SOCKETS]={0};
22.     char socketbuffer[NOOF_SOCKETS][100];
23.     int socketcounter=0;
24.     int highestsocket=0;
25. 
26.     struct sockaddr_in my_addr;
27. 
28.     int hsock;
29.     int * p_int ;
30. 
31.     socklen_t addr_size = 0;
32.     sockaddr_in sadr;
33. 
34.     fd_set  rFd, rFdx;
35.     fd_set  eFd, eFdx;
36.     fd_set  wFd, wFdx;
37. 
38. 
39.     hsock = socket(AF_INET, SOCK_STREAM, 0);
40.     if(hsock == -1){
41.         printf("Error initializing socket %d\n", errno);
42.         goto FINISH;
43.     }
44.     
45.     p_int = (int*)malloc(sizeof(int));
46.     *p_int = 1;
47.         
48.     if( (setsockopt(hsock, SOL_SOCKET, SO_REUSEADDR, (char*)p_int, sizeof(int)) == -1 )||
49.         (setsockopt(hsock, SOL_SOCKET, SO_KEEPALIVE, (char*)p_int, sizeof(int)) == -1 ) ){
50.         printf("Error setting options %d\n", errno);
51.         free(p_int);
52.         goto FINISH;
53.     }
54.     free(p_int);
55. 
56.     /* Set the socket in non blocking mode */
57.         if(fcntl(hsock, F_SETFL, O_NONBLOCK)==-1){
58.         printf("Error setting in non-block %d",errno );
59.         goto FINISH;
60.     }
61. 
62. 
63.     my_addr.sin_family = AF_INET ;
64.     my_addr.sin_port = htons(host_port);
65.     
66.     memset(&(my_addr.sin_zero), 0, 8);
67.     my_addr.sin_addr.s_addr = INADDR_ANY ;
68.     
69.     if( bind( hsock, (sockaddr*)&my_addr, sizeof(my_addr)) == -1 ){
70.         fprintf(stderr,"Error binding to socket, make sure nothing else is listening on this port %d\n",errno);
71.         goto FINISH;
72.     }
73.     if(listen( hsock, 10) == -1 ){
74.         fprintf(stderr, "Error listening %d\n",errno);
75.         goto FINISH;
76.     }
77. 
78.     //Now lets do the server stuff
79.     FD_ZERO(&rFd);
80.     FD_ZERO(&eFd);
81.     FD_ZERO(&wFd);
82. 
83.     addr_size = sizeof(sockaddr_in);
84. 
85.     int retv;
86.     int csock;
87. 
88.     FD_SET(hsock, &rFd);
89.     FD_SET(hsock, &eFd);
90.     FD_SET(hsock, &wFd);
91. 
92.     while(true){
93. 
94.         memcpy(&rFdx, &rFd, sizeof(rFd));
95.                 memcpy(&wFdx, &wFd, sizeof(wFd));
96.         memcpy(&eFdx, &eFd, sizeof(eFd));
97. 
98.         if( (retv = select((hsock<highestsocket?highestsocket:hsock)+1, &rFdx, &wFdx, &eFdx, 0) ) > 0){
99.             if(FD_ISSET(hsock, &rFdx)){
100.                 if((csock = accept( hsock, (sockaddr*)&sadr, &addr_size))!= -1){
101.                     printf("---------------------\nReceived connection from %s\n",inet_ntoa(sadr.sin_addr));
102.                     socketlist[socketcounter++]= csock;
103.                     if(csock > highestsocket)
104.                            highestsocket=csock;    
105.                     FD_SET(csock, &rFd);
106.                     FD_SET(csock, &eFd);
107.                 }
108.             }
109.             for(i=0;i<socketcounter;i++){
110.                 if(socketlist[i]){
111.                      if(FD_ISSET(socketlist[i], &rFdx)){
112.                         int recvbytes;
113.                         switch(recvbytes=recv(socketlist[i], socketbuffer[i], 100,0)){
114.                         case -1: 
115.                             printf("Read Error\n");
116.                             FD_CLR(socketlist[i], &rFd);
117.                             FD_CLR(socketlist[i], &eFd);
118.                             socketlist[i]=0;
119.                             break;
120.                         case 0:
121.                             printf("Socket closed\n");
122.                             FD_CLR(socketlist[i], &rFd);
123.                             FD_CLR(socketlist[i], &eFd);
124.                             socketlist[i]=0;
125.                             break;
126.                         default:
127.                             printf( "Read %d bytes from %d\n", recvbytes, socketlist[i]);    
128.                             socketbuffer[i][recvbytes]='\0';
129.                             FD_SET(socketlist[i], &wFd);
130.                         }
131.                     }
132.                      if(FD_ISSET(socketlist[i], &wFdx)){
133.                         int writebytes;
134.                         if((writebytes=send(socketlist[i], socketbuffer[i], strlen(socketbuffer[i]), 0))>0){
135.                             printf("Sent %d bytes to %d\n", writebytes, socketlist[i]);
136.                             FD_CLR(socketlist[i], &wFd);
137.                         }
138.                         else{
139.                             printf("Write Error\n");
140.                             FD_CLR(socketlist[i], &rFd);
141.                             FD_CLR(socketlist[i], &wFd);
142.                             FD_CLR(socketlist[i], &eFd);
143.                             socketlist[i]=0;
144.                         }
145.                     }
146.                 }
147.             }
148.         }
149.     }
150.     
151. FINISH:
152. ;
153. }

Hide line numbers

Save this file as OverlappedServer.cpp.

Now compile the file you created in step 1.

..workspace\SocketExample>>g++ OverlappedServer.cpp -lsocket -g -o server -lnsl

Write and compile the Client

The client reads a line from the console and sends this to the server. It then reads from the server and displays it on the console. Save this code as LinClient.cpp.

 1. #include <fcntl.h>
 2. #include <string.h>
 3. #include <stdlib.h>
 4. #include <errno.h>
 5. #include <stdio.h>
 6. #include <netinet/in.h>
 7. #include <resolv.h>
 8. #include <sys/socket.h>
 9. #include <arpa/inet.h>
10. #include <unistd.h>
11. 
12. int main(int argv, char** argc){
13. 
14.     int host_port= 1101;
15.     char* host_name="127.0.0.1";
16. 
17.     struct sockaddr_in my_addr;
18. 
19.     char buffer[1024];
20.     int bytecount;
21.     int buffer_len=0;
22. 
23.     int hsock;
24.     int * p_int;
25.     int err;
26. 
27.     hsock = socket(AF_INET, SOCK_STREAM, 0);
28.     if(hsock == -1){
29.         printf("Error initializing socket %d\n",errno);
30.         goto FINISH;
31.     }
32.     
33.     p_int = (int*)malloc(sizeof(int));
34.     *p_int = 1;
35.         
36.     if( (setsockopt(hsock, SOL_SOCKET, SO_REUSEADDR, (char*)p_int, sizeof(int)) == -1 )||
37.         (setsockopt(hsock, SOL_SOCKET, SO_KEEPALIVE, (char*)p_int, sizeof(int)) == -1 ) ){
38.         printf("Error setting options %d\n",errno);
39.         free(p_int);
40.         goto FINISH;
41.     }
42.     free(p_int);
43. 
44.     my_addr.sin_family = AF_INET ;
45.     my_addr.sin_port = htons(host_port);
46.     
47.     memset(&(my_addr.sin_zero), 0, 8);
48.     my_addr.sin_addr.s_addr = inet_addr(host_name);
49. 
50.     if( connect( hsock, (struct sockaddr*)&my_addr, sizeof(my_addr)) == -1 ){
51.         if((err = errno) != EINPROGRESS){
52.             fprintf(stderr, "Error connecting socket %d\n", errno);
53.             goto FINISH;
54.         }
55.     }
56. 
57.     //Now lets do the client related stuff
58. 
59.     buffer_len = 1024;
60. 
61.     memset(buffer, '\0', buffer_len);
62. 
63.     printf("Enter some text to send to the server (press enter)\n");
64.     fgets(buffer, 1024, stdin);
65.     buffer[strlen(buffer)-1]='\0';
66.     
67.     if( (bytecount=send(hsock, buffer, strlen(buffer),0))== -1){
68.         fprintf(stderr, "Error sending data %d\n", errno);
69.         goto FINISH;
70.     }
71.     printf("Sent bytes %d\n", bytecount);
72. 
73.     if((bytecount = recv(hsock, buffer, buffer_len, 0))== -1){
74.         fprintf(stderr, "Error receiving data %d\n", errno);
75.         goto FINISH;
76.     }
77.     printf("Recieved bytes %d\nReceived string \"%s\"\n", bytecount, buffer);
78. 
79.     close(hsock);
80.     
81. FINISH:
82. ;
83. }

Hide line numbers

Open a prompt to the working directory and compile the code using your C++ compiler

..workspace\SocketExample>g++ -o client LinClient.cpp

Run the example

Open a prompt to the working directory and run the server.

..workspace\SocketExample>server
Open another prompt to the working directory and run the Client.

..workspace\SocketExample>client
Now type something into the client prompt and press ENTER
You will see the exchange between client and server. You can open many client prompts at the same time and test.

C/C++ Client Server Socket Examples

noreply@blogger.com (righteous) — Fri, 15 Jun 2012 02:41:00 +0000

When talking about C/C++ client server examples, there are primarily three ways of handling multiple simultaeneous connections. You can spawn a new process using fork, creat a new thread using the POSIX thread library (or some other library) or you can use Overlapped IO. Our examples will cover all three options. These working examples are simple to understand, build and test.

Linux C++ Forking Server and Client Socket Example

Linux C++ Socket Example with Client Server and Mulit-Threading

Linux C++ Overlapped Server and Client Socket Example

Of course you can always use a combination of these methods for some truly awesome servers. But once you know the basics of threading, forking and overlapped io, you can be server programming guru.

Also we stronly suggest that you attempt the pthread model first, then the forked model and finally the overlapped model. All three examples share the client code.

You can download the all the 3 zipped example here.

LDAP: No structuralObjectClass for entry - Solved

noreply@blogger.com (righteous) — Sun, 10 Jun 2012 02:56:00 +0000

This error occurs when you try to add an entry into ldap. The error is something along the lines of no structuralobectclass or invalid structuralobectclassetc.

adding new entry
dn: uid=hello@hello.com,ou=users,o=xforce,o=au
ldap_add: Unknown error
ldap_add: additional info: no structuralObjectClass operational attribute

For example if your entry is something like the one below and keeps failing with the structural object class error..
dn: uid=hello@hello.com,ou=users,o=xforce,o=au
objectclass: person
uid=hello@hello.com
cn: hello
sn: world

Then you have to explicity add the structural object classattribute.

dn: uid=hello@hello.com,ou=users,o=xforce,o=au

objectclass: person

structuralObjectClass: person

uid=hello@hello.com

cn: hello

sn: world

LDAP: Shadow Context Error - Solved

noreply@blogger.com (righteous) — Sun, 10 Jun 2012 02:31:00 +0000

This error, typically takes the format of shadow context; no update referral. It occurs when you try to modify information in ldap and occurs even for the root user cn=manager.

adding new entry uid=hello@hello.com,ou=users,o=xforce,c=au
ldap_add: DSA is unwilling to perform
ldap_add: additional info: shadow context; no update referral

LDAP: error code 53 - shadow context; no update referral

This happens because the ldap is acting as a slave and will only allow updates form the 'dpdatedn' parameter in the ldap.conf file.

If you wish to make changes as root then you have to edit the ldap.conf file comment out all the lines with updated or add the manager as an updated and restart ldap.

C/C++ Regex Replace Function

noreply@blogger.com (righteous) — Thu, 19 Jan 2012 09:47:00 +0000

The aim is to implement a regex replace function that works in C/C++. Similar to php's preg_replace and perl's '=~'. The function is called regexreplace and supports groupings and back references. This function uses the regexec and regcomp functions, is simple easy to use and doesn't need Boost! One C/C++ function that does regular expression matching and replacement.

int regexreplace(char *pattern, char* replacement, char* instring, char** outstring, int maxlength);

It accepts a pattern to be matched, the replacement, the input string a char** outstring which will contain the result and the maxlength is the limit of the outstring.

Here is an example of how it can be used. Note the #include "regexreplace.h" that is required as the functions are defined in there.

 1. #include <stdlib.h>
 2. #include <stdio.h>
 3. #include <string.h>
 4. 
 5. #include "regexreplace.h"
 6. 
 7. int main(int vv, char** cc){
 8. 
 9.     char* result;
10. 
11.     if(regexreplace("shaped_\\([0-9]*\\)_\\([0-9]*\\)", "quota_policer(\\1,\\2)", "shaped_500_400  and shaped_900_100", &result, 256)==0){
12.         printf("====>%s<===\n",result); 
13.         free(result);
14.     }
15. 
16.     if(regexreplace("\\([0-9][0-9]*\\)_\\([abc]\\)", "quota_policer(\\1,\\2)", "hello_100_c mr barbie 987_a naf", &result, 256)==0){
17.         printf("====>%s<===\n",result); 
18.         free(result);
19.     }
20. 
21.     if(regexreplace("\\([0-9][0-9]*\\)_\\([abc]\\)", "quota_policer(\\1,\\2-\\3)", "hello_100_c mr barbie 987_a naf", &result, 256)==0){
22.         printf("====>%s<===\n",result); 
23.         free(result);
24.     }
25. 
26.     if(regexreplace("\\([0-9][0-9]*\\)_\\([abc]\\)", "", "hello_100_c mr barbie 987_a naf", &result, 256)==0){
27.         printf("====>%s<===\n",result); 
28.         free(result);
29.     }
30. 
31.     if(regexreplace("shaped_\\([0-9]*\\)_\\([0-9]*\\)", "quota_policer(\\1,\\2)", "shaped_500_400  and shaped_900_100", &result, 24)==0){
32.         printf("====>%s<===\n",result); 
33.         free(result);
34.     }
35. }

Hide line numbers

Below is the listing of the headerfile regexreplace.h. The idea was to keep it simple and not have to arrays and copy between arrays.

 1. #ifndef REGEXREPLACE_H
 2. #define REGEXREPLACE
 3. #include <string.h>
 4. #include <stdlib.h>
 5. #include <regex.h>
 6. 
 7. #define MAX_NMATCH    10
 8. typedef struct{
 9.     int offset;
10.     regmatch_t pmatch[MAX_NMATCH];
11. } regmatch_tt;
12. /* Move a string into a string */
13. int s1(char* s, int sl, char* r, int rl){
14.     memmove(s+rl, s+sl, strlen(s)-sl+1);
15.     memcpy(s, r, rl);
16.     return rl-sl;    
17. }
18. /* Handle realloc */
19. int reallocx(char** x, int s, int max){
20.     char *tmp;
21.     if(s>max)
22.         return -1;
23.     if((tmp = realloc(*x, s))!=0){
24.         *x = tmp;
25.         return 0; 
26.     }
27.     else
28.         return -1; 
29. } 
30. /* Use regexec, but to match all occurrences */
31. int regexec_matchall(regex_t* pregx, const char* instring, size_t nmatch_t, regmatch_tt * pmatcht, int flags ){
32.     int offset, counter;
33.     offset = 0;
34.     counter = 0;
35.     while (regexec (pregx, instring+offset, MAX_NMATCH, pmatcht[counter].pmatch, REG_EXTENDED)==0 && counter<nmatch_t) {  /* While matches found. */
36.         pmatcht[counter].offset = offset;
37.         offset+= pmatcht[counter].pmatch[0].rm_eo;
38.         counter++;
39.     }
40.     while(counter<nmatch_t-1){
41.         pmatcht[counter].offset=-1;
42.         counter++;
43.     }
44.     if(offset == 0)
45.         return -1;
46.     else
47.         return 0;
48. }
49. /* The function that actually does the work */
50. int preg_replace(regex_t* pregxo, char* replacement,  regex_t*  pregx, char* tomatchx, char** out, int maxlength){
51. 
52.     int i,j,k, offset, offset2; 
53. 
54.     regmatch_tt pmatcht[MAX_NMATCH];
55.     regmatch_tt pmatchto[MAX_NMATCH];
56. 
57.     regmatch_tt* pt, *mt, *xt;
58.     regmatch_t * p,  *m,  *x;
59. 
60.     char    tmpvalue2[256];
61.     char    *c;
62.     char    *tomatch = 0;
63.     int    deltalen;
64.     char    *tmpreplacement = 0;
65. 
66.     if(reallocx(&tomatch, strlen(tomatchx)+1,maxlength)==-1)
67.         goto FINISHED;
68.     else
69.         strcpy(tomatch, tomatchx);
70. 
71. 
72.     if(regexec_matchall(pregxo, replacement, 10, pmatchto, 0)!=0){
73.     }
74.     if(regexec_matchall(pregx, tomatch, 10, pmatcht, 0)!=0){
75.         return -1;
76.     }
77. 
78.     offset=0;
79. 
80.     for(i = 0 ; i < MAX_NMATCH &&  pmatcht[i].offset!=-1; i++){
81.         if(reallocx(&tmpreplacement, strlen(replacement)+1,maxlength)==-1)
82.             goto FINISHED;
83.         else
84.             strcpy(tmpreplacement, replacement);
85.         offset2=0;
86. 
87.         for( k = 0 ;  k < MAX_NMATCH && pmatchto[k].offset !=-1 ;k ++){
88.             pt = &pmatchto[k];
89.             p = &pt->pmatch[0];
90.             
91.             memset(tmpvalue2, 0, 256);
92.             strncpy(tmpvalue2, &tmpreplacement[p->rm_so+pt->offset+offset2], p->rm_eo-p->rm_so);
93. 
94.             while((c=strchr(tmpvalue2, '\\'))!=0)
95.                 *c='0';
96.             j = atoi(tmpvalue2);
97.             if(j< MAX_NMATCH && pmatcht[i].pmatch[j].rm_so !=-1 ){
98.                 mt = &pmatcht[i];
99.                 m = &mt->pmatch[j];
100.                 deltalen=(m->rm_eo-m->rm_so)-(p->rm_eo-p->rm_so);
101.                 if(deltalen>0 && reallocx(&tmpreplacement,strlen(tmpreplacement)+deltalen+1, maxlength)==-1)
102.                     goto FINISHED;
103.                 offset2+=s1(&tmpreplacement[p->rm_so+pt->offset+offset2], p->rm_eo-p->rm_so, 
104.                         &tomatch[m->rm_so+mt->offset+offset], m->rm_eo-m->rm_so);
105.             }
106.             else{
107.                 char* x ="";
108.                 offset2+=s1(&tmpreplacement[p->rm_so+pt->offset+offset2], p->rm_eo-p->rm_so, x, 0);
109.             }
110.         }    
111.         xt = &pmatcht[i];
112.         x = &xt->pmatch[0];
113. 
114.         deltalen=(strlen(tmpreplacement)-(x->rm_eo-x->rm_so));
115.         if(deltalen> 0 && reallocx(&tomatch, strlen(tomatch)+deltalen+1,maxlength)==-1)
116.             goto FINISHED;
117.         offset+=s1(&tomatch[x->rm_so+xt->offset+offset], x->rm_eo-x->rm_so, tmpreplacement, strlen(tmpreplacement ) );
118.         free(tmpreplacement);
119.         tmpreplacement=0;
120. 
121.     }
122.     *out=tomatch;
123.     return 0;
124. FINISHED:
125.     if(tomatch)
126.         free(tomatch);
127.     if(tmpreplacement)
128.         free(tmpreplacement);
129.     return -1;
130. }
131. /* The regexec replace function */
132. int regexreplace(char * pattern, char* replacement, char* instring,  char** outstring, int maxlength){
133.     regex_t pregx, pregxo;
134.     int retvalue;
135.     const char* patterno = "\\(\\\\[0-9][0-9]*\\)";
136.     if(regcomp(&pregxo, patterno, 0)!=0){
137.         return -1; }
138.     if(regcomp(&pregx, pattern, 0)!=0){
139.         return -1; }
140.     retvalue=preg_replace(&pregxo, replacement,  &pregx, instring, outstring, maxlength);
141.     regfree(&pregxo);
142.     regfree(&pregx);
143.     return retvalue;
144. }
145. #endif

Hide line numbers

C/C++ Regular Expressions - Regex/Regexec

noreply@blogger.com (righteous) — Thu, 19 Jan 2012 04:06:00 +0000

Aim
The aim of this example is to demonstrate the regex functions available in C/C++. This example is not using Boost. This code snippet shows how you can use regcomp and regexec to match a string with grouping and backreferences. Please use this as an introduction into regular expressions for C/C++ (For a more sophisticated function that does replacement aswell pleas see C/C++ regexreplace).

 1. #include <stdio.h>
 2. #include <string.h>
 3. #include <regex.h>
 4. 
 5. int main(int vv, char** c){
 6. 
 7.     regex_t pregx;
 8.     regmatch_t pmatch[10];
 9.     int i;
10.     const char* pattern = "\\([0-9][0-9]*\\)_\\([0-9][0-9]*\\)";
11.     char tomatch[256] = "This is hello_100_200_";
12.     char tmpvalue[256];
13. 
14.     if(regcomp(&pregx, pattern, 0)!=0){
15.         printf("Pattern failed to compile\n");
16.         return -1;
17.     }
18.     if(regexec(&pregx, tomatch, 10, pmatch,0)!=0){
19.         printf("Could not match pattern %s  %s\n",pattern, tomatch);
20.         return -1;
21.     }
22. 
23.     for(i = 0 ; i < 10 && pmatch[i].rm_so!=-1 ; i++){
24.         memset(tmpvalue, 0, 256);
25.         strncpy(tmpvalue, &tomatch[pmatch[i].rm_so], pmatch[i].rm_eo-pmatch[i].rm_so);
26.         printf("matched pmatch[%d].rm_so: %d    pmatch[%d].rm_eo: %d  -   \"%s\"\n",
27.                 i, pmatch[i].rm_so, i, pmatch[i].rm_eo, tmpvalue);    
28.     }
29.     return 0;
30. }

Hide line numbers

One of the drawbacks of regexec is that it doesn't match multiple times. For example if my string was '
This is hello_100_200_ and 700_600' it would only match the 100_200.

..workspace\>gcc -c reg.c
..workspace\>./a.out
matched pmatch[0].rm_so: 14 pmatch[0].rm_eo: 21 - "100_200"
matched pmatch[1].rm_so: 14 pmatch[1].rm_eo: 17 - "100"
matched pmatch[2].rm_so: 18 pmatch[2].rm_eo: 21 - "200"

As you can see \\0 matches 100_200

Oracle TimesTen Getting Unixtime or Seconds Since the Epoch UTC

noreply@blogger.com (righteous) — Sat, 14 Jan 2012 02:18:00 +0000

On normal oracle databases, you can run a simple sql query subtracting the epoch date from sysdate. This doesn't work for TimesTen as you will get an error mentioning 'An interval data type must be specified for a datetime arithmetic result.' The easiest work around is to cast the subtraction as a bigint and divide by 1000000

On TimesTen - sql
select cast((sysdate - to_date('1970-01-01 00:00:00','YYYY-MM-DD HH24:MI:SS')) as bigint)/1000000 from dual

On conventional oracle database - sql
select (sysdate - to_date('1970-01-01 00:00:00','YYYY-MM-DD HH24:MI:SS')) * 86400 from dual

C/C++ Trimming whitespace - Trim

noreply@blogger.com (righteous) — Wed, 28 Dec 2011 05:42:00 +0000

How to trim strings in C/C++ is a commonly asked question. Below is a simple C/C++ function that will trim leading and trailing whitespaces. Whitespaces include spaces, tabs, carriage return, new lines etc. It get's rid of 'white' spaces at the beginning and at the end of an array of characters.

 1. int trim(char* c){
 2.         char * i;
 3.         if(c==0 || !strlen(c))
 4.                 return 0;
 5.         for(i=c ; isspace(*i)!=0 && (i-c)<strlen(c); i++);
 6.         memmove(c, i, strlen(c)-(i-c)+1);
 7.         if(!strlen(c))
 8.                 return 0;
 9.         for(i=(c+strlen(c)-1) ; isspace(*i)!=0 && i>c; i--);
10.         *(i+1)='\0';
11.         return strlen(c);
12. }
13. 
14. int main(int vv, char** cc){
15.         char  xman[] ="  \t String that needs trimming \n ";
16.         trim(xman);
17.         printf ("--%s--\n", xman);
18. }

Hide line numbers

Image Compression Using Php

noreply@blogger.com (righteous) — Thu, 17 Nov 2011 10:09:00 +0000

Aim
Php has a fair amount of image manipulating functions, but none that really offer any kind of image compression. By image compression we mean sacrificing the quality of the image for faster load times by reducing the size of the image.

The aim is to make a fast and simple function, without the need for temporary files, and without the need to create additional temporary images for interpolation.

Method
The problem is that there is no way to get the 'quality' out of a jpeg, so we estimate the quality based on some assumptions. We also do not try to compress images that are less than 2Mb because the resulting files are usually bigger that the input files.

 1. <?php
 2. ini_set('display_errors', 1); 
 3. ini_set('log_errors', 1); 
 4. error_reporting(E_ALL);
 5. 
 6. function LossifyJpegImage(& $data){
 7.     $size=strlen($data);
 8.     if($size<2048){
 9.         echo $data;
10.     }
11.     else if(($im = imagecreatefromstring($data))!==FALSE){
12.         $x = imagesx ($im);
13.         $y = imagesy ($im);
14.         $ratio = intval(($size*100)/($y*$x));
15.         $quality = intval(log($ratio)/0.043);
16.         imagejpeg($im, NULL, intval($quality/2));
17.         file_put_contents('php://stderr', "size $size, assumed quality $quality\n");
18.         imagedestroy($im);
19.     }else{
20.         echo $data;
21.     }
22. }
23. 
24. $binary_data = file_get_contents($argv[1]);
25. 
26. LossifyJpegImage($binary_data);
27. ?>

Hide line numbers

Now run the sccript giving it a filename on the command line and pipe the output to a file.

..workspace>php compression.php oldfile.jpg > newfile.jpg

The algorithm is modelled very simply and it is based on empirical tests. It is assumed that it holds true for most images. It is based on this equation that the quality of the image is ascertained.

Comment on Free Proxies

noreply@blogger.com (righteous) — Sun, 16 Oct 2011 00:51:00 +0000

Please feel free to leave any kind of comment. Helpful or not. Maybe you love our proxy, maybe you think it really stinks, we don't care what you think, just leave us any kind of comment and we will delete the unking ones :).

Please state which proxy you were using - web proxy or the ip:port proxy.

Thanks!

STL Starter 1 - User Defined Predicate Functions (Standard Template LIbrary)

noreply@blogger.com (righteous) — Tue, 16 Aug 2011 02:02:00 +0000

Aim
The aim of this example is to show how you can use user defined binary and unary predicate functions (not functors) and the bind2nd with the Standard Template Library. We will be using a vector for this purpose, but it applies for all containers. In this example we use a class that represents a complex number. Complex numbers are compared by squaring both real and imaginary components and adding them together.

Line 50 Use our user defined ValueGreaterThan20 unary predicate in the find_if

Line 54 Use the ValueGreaterThan Binary predicate, but we use bind2nd to make it unary.

Line 57 Use the binary predicate SortValueGreaerThan in the sort function.

Do not forget to use ptr_fun when passing the function pointers to the bind2nd. Ptr_fun takes a function pointer as its argument and returns a function pointer adapter. If you forget you will get some errors such as below
../include/c++/backward/binders.h:138: error: 'bool ()(ComplexNumber, int)' is not a class, struct, or union type

 1. #include <vector>
 2. #include <functional>
 3. #include <algorithm>
 4. #include <iostream>
 5. 
 6. using namespace std;
 7. 
 8. template <class Operation, class T>
 9.   bool ninjax (const Operation& op, const T& x)
10. {
11.     return true;
12.  }
13. 
14. class ComplexNumber{
15. public:
16.     ComplexNumber(int r, int i): real(r), imaginery(i){}
17.     int real;
18.     int imaginery;
19. };
20. 
21. //Unary predicate
22. bool ValueGreaterThan20(ComplexNumber c){
23.     return c.imaginery*c.imaginery+c.real*c.real > 20*20; 
24. }
25. 
26. //Binary predicate
27. bool ValueGreaterThan(ComplexNumber c, int i){
28.     cout<<c.imaginery<<endl;
29.     return c.imaginery*c.imaginery+c.real*c.real > i*i;
30. }
31. 
32. //Binary predicate
33. bool SortValueGreaterThan(ComplexNumber c, ComplexNumber b){
34.     cout<<c.imaginery<<endl;
35.     return c.imaginery*c.imaginery+c.real*c.real > b.imaginery*b.imaginery+b.real*b.real;
36. }
37. 
38. int main(){
39. 
40.     vector<ComplexNumber> complexNumbersVec;
41. 
42.     complexNumbersVec.push_back(ComplexNumber(5,5));
43.     complexNumbersVec.push_back(ComplexNumber(10,15));
44.     complexNumbersVec.push_back(ComplexNumber(15,5));
45. 
46.     complexNumbersVec.push_back(ComplexNumber(10,20));
47. 
48.     //Using bind2nd with user defined unary predicate function
49.     vector< ComplexNumber >::iterator location =    
50.         find_if(complexNumbersVec.begin(), complexNumbersVec.end(), ValueGreaterThan20);
51. 
52.     //Using bind2nd with user defined binary predicate function
53.     vector< ComplexNumber >::iterator location2 =    
54.         find_if(complexNumbersVec.begin(), complexNumbersVec.end(), bind2nd(ptr_fun(ValueGreaterThan), 0));
55. 
56.     //Using sort with user defined binary predicate funcion
57.     sort (complexNumbersVec.begin(), complexNumbersVec.end(), SortValueGreaterThan);
58. 
59.     for (location=complexNumbersVec.begin(); location!=complexNumbersVec.end(); location++)
60.         cout << " " << location->real <<", "<<location->imaginery<<"i"<<endl;
61. 
62. }

Hide line numbers

Check Page Rank of all Blog Posts

noreply@blogger.com (righteous) — Thu, 14 Jul 2011 09:04:00 +0000

Here is a nifty tool to ffind out all the page ranks of your individual blog posts. All you have to feed it is your blog's atom or rss feed and the tool will display the PR of all your posts.

For wordpress blogs the RSS feed is "/feed" after your blog. ex: http://myblog.wordpress.com/feed

Automatically Create Sitemaps for Blogs

noreply@blogger.com (righteous) — Thu, 14 Jul 2011 04:25:00 +0000

Automatically create sitemaps for all your blogs by enteirn the atom or RSS feeds. You can create a sitemap in HTML for search bots. In human readable form or even XML using the sitemap protocol.

Compare Page Ranks of Multiple Sites

noreply@blogger.com (righteous) — Thu, 14 Jul 2011 04:14:00 +0000

Use this tool to compare the page ranks of multiple sites. No messy capthas. Quickly and easily find out the PR of multiple sites.

C/C++ Forking and Using Pipes to Exchange Data Between Parent and Child

noreply@blogger.com (righteous) — Tue, 29 Mar 2011 04:05:00 +0000

Aim
Pipes are form of inter-process communication. They are very useful to synchronize access to resources, notification and exchanging data. In this example we will use fork to spawn a process and then communicate between the child process and parent process using pipes.

Assumptions
You have a C++ compiler (g++ in this example) and access to a linux environment or cygwin.

Application
In this simple example we use pipe to create the pipes. We then fork and use write and read on the pipes to exchange information. Also note that we call waitpid in the parent to stop it exiting before the child has quit.

Let us have a look at the code below. This file will be saved as forking_pipes.cpp

 1. #include <stdlib.h>
 2. #include <stdio.h>
 3. #include <string.h>
 4. #include <unistd.h>
 5. #include <errno.h>
 6. #include <sys/types.h>
 7. #include <sys/wait.h>
 8. 
 9. 
10. int main(int argv, char** argc){
11. 
12.     int pipes[2];
13.     char readbuf[64]={0};
14.     char writebuf[64]={0};
15.     int childpid;
16. 
17.     if(pipe(pipes)<0){
18.         perror("Pipe allocation failed");
19.         exit(0);
20.     }
21. 
22.     switch(childpid=fork()){
23.         case -1:
24.             perror("Problem with fork\n");
25.             break;
26. 
27.         case 0:  //Child
28.             //Writing
29.             sprintf(writebuf, "Hello from %d", getpid());
30.             if(write(pipes[1], writebuf, strlen(writebuf))<0){
31.                 perror("Child write failed\n");
32.             }
33. 
34.             //Reading
35.             if(read(pipes[0], readbuf, 64)>0){
36.                 printf("Child read - %s\n", readbuf);
37.             }
38.             else{
39.                 perror("child read failed\n");
40.             }
41. 
42.             break;
43.         
44.         default: //Parent
45.             //Reading
46.             if(read(pipes[0], readbuf, 64)>0){
47.                 printf("Parent read - %s\n", readbuf);
48.             }
49.             else{
50.                 perror("Parent read failed\n");
51.             }
52.             
53.             //Writing
54.             sprintf(writebuf, "Hello from parent");
55.             if(write(pipes[1], writebuf, strlen(writebuf))<0){
56.                 perror("Parent write failed\n");
57.             }
58. 
59.             waitpid(childpid,0,0);
60. 
61.             break;
62.     }
63. }

Hide line numbers

Compile the code

..workspace\Forkingpipes>g++ forking_pipes.cpp

pipes[0] is the reading pipe and pipe[1] is the writing pipe. In the parent whatever you write to pipe[1] can be read at the child using pipe[0]. In the child whatever you write to pipe[1] can be read at the Parent using pipe[0].

If you wish to spawn multiple children then you have to have multiple pipe pairs. This means you will have to use select in the parent to find out which pipes are ready for writing and reading.

Bandwidth Throttling and Connection Limiting for Web, Proxy and Email Servers

noreply@blogger.com (righteous) — Fri, 18 Mar 2011 03:29:00 +0000

Aim
The aim of this program is to create a self regulating tcp/ip connection forwarding program that can limit connections and do per IP bandwidth throttling and connection limiting. This can be used to give bandwidth throttling and connection limiting to proxy, email, web and ftp servers. It will work with any server such as tomcat, apache, glassfish, squid, weblogic etc as it runs as a separate standalone application. This software is now called SpeedBump. This can be easily used to throttle Apache and Squid very easily.

Resources

Project Kimbula. - Project hosted on google's code hosting
Download Source Now - Download source code.
Installation

The key features are

Set the maximum number of con-current connections. Similar to apache's MaxClients.
Set maximum con-current connections per ip. Similar to conn-limit in iptables.
Set the new connection burst and average connections per second per ip. Similar to limit-burst in iptables.
Set the uplink and downlink throughput per ip. The throughput will be divided amongst the concurrent connections per ip.
Once a certain amount of bytes has been uploaded/downloaded drop the available throughput for a time period.
Written in simple, easy to understand C.
Add and remove headers for upstream and downstream connections

Let us look at a typical scenario below. One user has opened up a lot of connections to the server and is downloading or uploading a lot of data.

The server now has less connections available for the other users and the most of the bandwidth is being used by the one greedy user. This usually results in a poor experience for other users.

Now we bring in the SpeedBump. This can be located on a seperate server or co-located with the existing server. For purposes of illustration let us show them on separate servers. Now all the connections have to first go through SpeedBump before they can get to the server.

SpeedBump will now reject all connections above the connection limit and throttle the bandwidth used by the greedy user. This reduces the load on the web/proxy server and gives a consistent experience to all the users.

Software Maintained by TidyTutorials

noreply@blogger.com (righteous) — Fri, 18 Mar 2011 03:11:00 +0000

Here is a list of the software currently maintained by TidyTutorials.

Free IP:Port Proxy Emulator
This emulates the behaviour of an ip:port or elite proxy, but is written in php and runs inside an Apache web server. The only ip:port proxy emulator able to push rated advertisements! Contains Turing Test. The source code and installation instruction can be found at Project Mugatiya.
Basic requirement is apache + php + should be able to configure .htaccess.
Hosting : VPS/VDS

Free Web Proxy
Yet another web proxy such as Glype or PHProxy. Display advertisements and has the ability do display rated advertisements. Has unique ability to be rented out! The source code and installation instruction can be found at Project Uguduwa.
Basic requirement is apache + php + shoulb be able to configure .htaccess.
Hosting: Shared or VPS/VDS

Free SpeedBump Software
SpeedBump runs in front of a proxy server or web server and regulates connections and throttles bandwitdh. Don't let a single IP hog all your connections and bandwith. Set max concurrent connections per ip, max burst and average connections per ip, througput per ip and many more. Can be modified to be used as a proxy. The source code and installation instruction can be found at Project Kimbula.
Basic requriement gcc/g++
Hosing: VPS/VDS

Common Database Pool Template
A common database pool using a template that will support Oracle, MySQL and SQLServer and use as common a process and methodology as possible to access connections. More information can be found here

Tracing Memory Leak in C/C++ Using Mtrace

noreply@blogger.com (righteous) — Fri, 04 Mar 2011 00:08:00 +0000

Aim
The aim of this tutorial is to show how you how to trace a memory leak in a C/C++ program using mcheck. This allows you to match your malloc, realloc, calloc etc to your frees. Mtrace functionality is available with the gnu C/C++ dev packages.

Let us take a look at the sample program below. Let's call this file leaker.c

 1. #include <stdlib.h>
 2. 
 3. #include <mcheck.h>
 4. 
 5. 
 6. int main(int argv, char** argc){
 7.         mtrace();
 8.         char * c1 = malloc(10*sizeof(char));
 9.         char * c2 = malloc(10*sizeof(char));
10. 
11.         c1[2]='a';
12.         c2[2]='b';
13. 
14.         free(c2);
15. 
16.         muntrace();
17. }

Hide line numbers

Line 3 You need to include the mcheck.h header file

Line 7 call mtrace() to start the tracing.

Line 16 call muntrace to stop the tracing.

As you can see this program has a blatant memory leak, in that we are not freeing pointer C2. Next we have to compile the program with the -g switch so that we can find out at which line the leak is occurring.

gcc -g leaker.c

This will create the default a.out executable. Before we run the program we need to set an environment variable called MALLOC_TRACE. This will contain the location of the tracefile to be created by mtrace/mcheck. The example below is for the bash shell.

>MALLOC_TRACE=/home/righteous/mt.txt

Now we run the program. Once the program has run it's course you can take a look at the trace file that it has created.

>./a.out
>
>cat mt.txt
= Start
@ ./a.out:[0x8048456] + 0x8968378 0xa
@ ./a.out:[0x8048465] + 0x8968388 0xa
@ ./a.out:[0x8048485] - 0x8968388
= End

To make sense of this file we need another command called mtrace. This is a perl script. This may or may not be available in your environment. You can search for the script and download it.
Now we run the script and you will see details of the memory leak below.

>mtrace a.out mt.txt

Memory not freed:
-----------------
Address Size Caller
0x08968378 0xa at /home/test.c:8 ??:0

Things get a lot trickier in an environment where you are forking process. For instance if you allocate memory, fork and then delete the memory in the child process, this will be flagged as an error as the child never actually allocated the memory. Also if you start mtrace after you have allocated memory or call muntrace before you have freed all your memory, mtrace will highlight these as errors.

Apache/Php Web Proxy

noreply@blogger.com (righteous) — Fri, 17 Dec 2010 04:46:00 +0000

Aim
The aim of this exercise is to create yet another web proxy. This is a url rewriting proxy so the user experience is not the very best. It is written as a Php script, it will run on shared hosting. There is no database requirement.

As this is a fairly substantial project, it is hosted on google at Project uguduwa. There is also more information at the Wordpress blog projectuguduwa.blogspot.com.

The advantages

Run in shared hosting or Virtual Private or Dedicated server
Modify content to display adds
Has a content filter
Remote Ip, requested url and requested hostname based black and white lists
Written in simple, easy to understant Php.

Resources

Project Uguduwa. - Project hosted on google's code hosting
Download Source Now - Download source code and zip file
projectuguduwa.blogspot.com. For installation, configuration and other relevant information

Apache/Php IP:Port Proxy Emulator

noreply@blogger.com (righteous) — Wed, 17 Nov 2010 01:53:00 +0000

Aim
The aim of this exercise is to create an IP:Port proxy emulator in Php that can be deployed to a web server such as Apache. Once deployed it will run almost exactly like an IP:Port proxy -also referred to as Elite or Open proxies- requring you to change your browser settings. Because it's written as a Php script, it will run inside apace. There is no database requirement.

As this is a fairly substantial project, it is hosted on google at Project Mugatiya. There is also more information at the the 'Project Mugatiya blog Project Mugatiya on TidyTutorials.

The advantages

Almost exactly like a proxy such as squid or mod_proxy.
Works with flash, streaming, torrents, pretty much everything.
Modify content to display adds
Has inbuilt Turing Test (reCaptcha) to keep away bots
Has a content filter
Remote Ip, requested url and requested hostname based black and white lists
Written in simple, easy to understand Php.

Resources

Project Mugatiya. - Project hosted on google's code hosting
Download Source Now - Download source code and zip file
projectmugatiya. For installation, configuration and other relevant information

You should note that the proxy does rewrite HTTPS requests as plain HTTP requests and this sometimes doesn't work with complex sites. Please feel free to contribute to the project.

Also, remember that as this is an IP:Port proxy, if left unchecked could land you with exorbitant overages on bandwidth. This demolishes web proxies hands down.

Using Tools such as AWK, GREP, NAWK, XARGs on the Command Line

noreply@blogger.com (righteous) — Tue, 26 Oct 2010 02:15:00 +0000

Extract the requests from an access_log that match the IPs from a list.

We have your common log formatted access_log. We wish to get all the requests that match the IPs from a list of IPs in a text file called ip_list.txt.

# awk '/.*/ {print "cat access_log | grep "$1" >> extract.txt "}' ip_list.txt | sh

Find out the Files That Contain Certain Text

It is common to want all the names of the files that contain a certain text. For instance, say inside a script directory you want to find all the scripts that contain a certain database name or a piece of text.

find . -name '*.sh' | xargs grep -il '_KingKongDB'

Find out required libraries for C/C++ functions

Here we want to use the function sem_init, but we are not too sure of what libraries to link with. We can use 'nm' and 'grep' on the libraries (/usr/lib on linux).

# nm --print-file-name /usr/lib/* 2> /dev/null |grep sem_init

This command will return results with strings containing '/usr/lib/libpthread.a'. From this you know that you have to link with the option -pthread.

Replace a value in multiple files and write the output files to a new directory

In this example we have xml files such as Nokia-220s.xml, Nokia-2330.xml etc. We want to replace the value 'Spec-Handler' in these files with a new value called 'Nokia-WAP' and then write these files into a new directory called 'new'

# ls -l Nokia* |awk '{print "sed","'\''s/Spec-Handset/Nokia-WAP/g'\'' " $9 " >new/"$9}' |sh

Replace the newline ('\n') character at the end of a line with something else (a comma in this case)

cat addresses.txt | tr '\n' ','

Compare two files that are similar and list values in one file that do not appear in the other

cat file1 | awk '{print"echo " $2 " `grep -c " $2" file2`"}' | sh | grep '0$'

Find the Min and Max in a list of values

For example if your file contents look like below
11
22
33
44

You can calculate min/max/average using nawk

cat t | nawk '{print $1}' | nawk 'NR==1 {min=$1;max=$1;sum+=$1;next} {if ($1max) max=$1;sum+=$1} END{printf "min: %0.2f max: %0.2f avg: %0.2f\n", min,max,sum/NR}'

min: 11.00 max: 44.00 avg: 27.5

Misc

find . -type f | grep -v '\.svn' | xargs perl -pi -e 's/\r\n/\n/g'

cat access_log | grep 217.124.181.30 |sed -n 's/.*$"[a-zA-Z][^"]*"$$/\1/pg'

Break Out of Web Proxy Frames

noreply@blogger.com (righteous) — Mon, 11 Oct 2010 01:44:00 +0000

Aim
To break out of Web Proxy Frames or any frames for that matter all you have to do is add the javascript shown below to your web page and replace the codediaries.com/break.html link with your own link. Web Proxies can sometimes be a nuisance to webmasters as these proxies sometimes stop adds and mangle content. With this code snippet, you can easily break out of 98% of Web Proxies, provided they have javascript turned on. Let's bust all those web proxies!

 1. <script>
 2. if (top.location!=("http://codediaries.com/break.html")) {
 3.             top.location = "http://codediaries.com/break.html";
 4. }
 5. </script>

Hide line numbers

To see this in action visit the link http://codediaries.com/break.html using a web proxy.

Unfortunately Web Proxies are a little smarter than your average Frame and they tend to rewrite and insert a lot of javascript code. You get around this by obfuscating the javascript code using any online obfuscater. Once done, the code will look like this.

 1. <script>
 2. var _0x5135=["\x6C\x6F\x63\x61\x74\x69\x6F\x6E","\x68\x74\x74\x70\x3A\x2F\x2F\x63\x6F\x64\x65\x64\x69\x61\x72\x69\x65\x73\x2E\x63\x6F\x6D\x2F\x62\x72\x65\x61\x6B\x2E\x68\x74\x6D\x6C"];if(top[_0x5135[0]]!=(_0x5135[1])){top[_0x5135[0]]=_0x5135[1];} ;
 3. </script>

Hide line numbers

Javascript obfuscators are readily available and most of them are online. Just search for "javascript obfuscator" in your favourite search engine.

Once you learn this simple trick you can now make changes and experiment on breaking these Web Proxies and Frames. Use this as a starting point for your web proxy demolition.

Tomcat Configuration - Home, Base and catalina.policy

noreply@blogger.com (righteous) — Mon, 27 Sep 2010 03:04:00 +0000

Here is some generic information regarding Tomcat 5.5 configuration and giving read write execute permissions to a servlet. Java's security manager can be quite complex and usually leads to a lot of pain when configuring a servlet to do something that is above and beyond the standard "read", "write", "execute" such as delete. Debugging such Security Manager permission error can be fairly daunting.

Tomcat configuration has two variables catalina.home and catalina.base. The purpose of this is to allow multiple instances of tomcat run from a single installation.

catalina.home is the installation directory and contains the directories bin, common, server.

catalina.base is your instance directory and contains the instance directories such as work, shared, conf, webapps.

Security

In your conf directory there is another directory called policy.d. Inside the policy.d directory there are a few files. All of these files are merged into one and written to the catalina.policy file in the conf directory. DO NOT edit the catalina.policy file as this will have no effect. You have to edit the files inside the policy.d directory.

To give your servlet extra permissions edit the file 50users.policy and add your servlet

 1. grant codeBase "file:${catalina.base}/webapps/Android/WEB-INF/-" {
 2.  permission java.io.FilePermission "${catalina.base}/webapps/Android/-", "read,write,execute,delete";
 3. };

Hide line numbers

You must be careful here. If, for example you are using another package to modify some files such as the DeleteQuietly function in the io-commons.jar package, then you must give this jar file the permissions too, if it is in another directory.

Debugging Security

To debug you must start tomcat with the option -Djava.security.manager -Djava.security.debug=all

If you are using the startup script in /etc/init.d/tomcat5.5 then you should edit the tomcat5.5 file and add the setting as shown below.

 1. if [ "$TOMCAT5_SECURITY" = "yes" ]; then
 2.         JAVA_OPTS="$JAVA_OPTS  -Djava.security.manager -Djava.security.debug=all  -Djava.security.policy=$CATALINA_BASE/conf/catalina.policy"
 3. fi

Hide line numbers

This will give you a huge amount of logging info, but you can easily search the output for something like 'access denied' to find out what is causing the problem