In this talk, we will discuss what technical debt is, when it is acceptable and when it isn’t, and strategies for effectively managing it, both on an independent and team level. We’ll talk about how it affects the entire team and how to work through minimizing it, as well as working effectively as a team to address it.

(see also the slides from this talk: http://naramore.net/slides/DPC10-techdebt.pdf)

APC and Memcache both provide effective mechanism for caching data within PHP applications to improve performance and latency. This talk will cover when to use both of these tools to get the most of your apps and which workflows better suit each tool.

While this is an important steps toward application interoperability, there is often an important step missing. Most interoperable applications lack one final feature that allows full seamless integration: data sourcing, or the ability to get the data it needs from elsewhere.

With data sourcing, we are not just importing data into our applications, we are using outside sources as the source for that data without creating redundancy. A simple example is the data sourcing of the user information. Most applications have their own user table. Applications that feature data sourcing of users make it possible to tell the system to get the user data not from its internal database but from a different source, for example the database of another application, an LDAP server or a web service that provides the user data. In the case of user data, if you have five applications that each have a database of users, it would be a lot simpler to integrate those applications if you could use one of them as the master source for the user data and configure the others to refer to it.

The principle is applicable to more than just users. Groups comes to mind (particularly groups within an organization that you may want to use within your applications), and friends is another common topic (aren’t you tired of befriending all your friends on every new social website?). In the case of ecommerce systems it would be great if you could use data sourcing to get the actual product data from different systems. Magento, the popular ecommerce application, has product import/export functionality, but there is no easy way to tell it to connect to a web service to get the data for products instead of looking at its own database. This makes it hard to plug Magento as an ecommerce module into a larger system; most implementations you will find have Magento at its core and other, more flexible, systems plugging into it.

Data sourcing can help applications such as Magento reach a wider audience and especially help it be used in enterprise scenarios where there are many components that together form a big system.

The concept of data sourcing is comparable to Dependency Injection; instead of hardwiring the dependencies within the software, you tell it its dependencies so that at runtime it can connect to the correct components and get its data.

Two Flavours of Data Sourcing

When you want to use data sourcing, there are in essence two ways to do this.

Synchronisation

The first one is synchronisation. This means that the data is still local to the application, but it is (periodically or on the fly) synchronized with external applications. For the application being plugged into a system, this generally means that hardly any modifications are necessary; a script needs to be written that simply synchronises the data between the sources.

While this works in some situations, it is undesirable in most situations. If you have multiple applications you will end up with multiple copies of the data, and run the risk of having data out of sync between the sources. It can also lead to ownership or privacy issues. One application should be the owner of a data set; if that data set is exported to other applications then you lose a certain amount of control over it.

Federation

The better option is federation. Federation basically means that you will get the data from its source when you need it. You could still cache for performance reason but there is no mass synchronisation going on. This is the method we will be looking at in the rest of this article, as it is the one that is the most interesting, but also the one that requires work in the applications that want to make use of federation.

Implementing Data Sourcing

Imagine you have built an application that shows you a person’s wishlist. You may have a query in there somewhere that joins the wishlist table with the user table and a few auxiliary tables with categorisation information. If the application is built like that, and you decide to install the application within an environment with multiple applications, it will be hard to make the application use user accounts from another application; you will have to not only rewrite all the queries, you will also have to find a way to connect the application data with the external user accounts. There are a few things that will help you make your application ready for data sourcing, so let’s take a look at them.

Models

If your application is set up according to the Model/View/Controller (MVC) paradigm, then your business logic is already isolated within your models. This will make it easier to source the data from outside sources. It’s a matter of taking the user model and changing that to get the users from elsewhere. Using models within an application is a recommendation that I would make regardless of whether you plan on doing data sourcing, but it is particularly helpful when abstracting data away from your core application. Here is a simple code snippet that illustrates the idea, without giving detail:

class UserModel
{
    ....
 
    public function getUser($id)
    {
       // Classic db code replaced by external data source
       // return db::getrows("select * from users where id=:1", $id); 
 
       return $this->getSoapClient()->getUser($id);
    }
}

In this example, the application will no longer fetch its data from the database, but use a SOAP client to retrieve the data. Of course it is very easy if the SOAP server has the exact same interface as our model, but that will rarely be the case when you’re talking to external services. If it differs, you may have to do some parameter and result set transformations, but you get the idea.

Protocol Abstraction

In the above example we are connecting to a SOAP service, which allows us to get the users from another application if it has a SOAP service, but what happens if your application is used in a scenario where there is no SOAP server, or the SOAP server uses a different set of methods? If we want to be flexible, we have to abstract the data source from our model. This is where the ‘Data Mapper’ design pattern, presented by Martin Fowler in his book Patterns of Enterprise Application Architecture, comes in handy. This pattern abstracts the data source away from the model. Our above example, adapted to include this idea, could look like this:

class UserModel
{
    ...
 
    public function getUser($id)
    {
         return $this->getDataMapper()->getUser($id);
    }
}

You could then provide several data mapper implementations for all the external sources your application supports:

class DataMapper_User_Mysql implements DataMapper_User { ... }
class DataMapper_User_SomeSOAPService 
                         extends DataMapper_User_SOAP_Abstract 
                         implements DataMapper_User { ... }

Note the use of ‘SomeSoapService’ in the name and not just ‘SOAP’. SOAP is a standard but every SOAP service can implement their own methods, so we would have to have a separate mappers for different services. To avoid reinventing the wheel across different SOAP services we can derive from a general abstract SOAP class that takes care of the SOAP bits that are standard across all SOAP services. Later we will see how REST and the use of standards make this easier.

REST

REST is a nicer protocol than SOAP because it’s not only easier to implement, it has a lot less overhead and is easier to standardize. Because REST basically uses resource urls and the HTTP verbs to operate on them, they make data sourcing easier. For example, consider the following resource urls:

• http://example.com/users
• http://example.com/users/42

The first resource is a collection of users. We can retrieve the users using an HTTP GET request, we can add users using HTTP PUT, etc. The second resource is an individual user, which we can retrieve by performing a GET request, delete by performing a DELETE request etc.

Applying this to your application might mean that in the end, all you need to do is add some configuration into your application, rather than needing to modify code, for example:

$datasources = array(
    'userservice' => 'SomePeopleService',
    'groupservice' => 'SomeGroupService');

Your REST data mapper then performs simple REST calls to retrieve it’s data through these services. Of course, you would still have to know what results to expect and how to map the results to your specific implementation, so you still might end up using multiple REST data mappers. The use of finer grained standards helps here. REST makes things easy; knowing that the response will be JSON will make things even easier; knowing that the JSON response is formatted according to the OpenSocial standard makes it peanuts. Which brings us to the final topic.

Standards

Federating access to applications for centrally managed users is very common. Large organisations deal with this all the time and nowadays, with so many social networking applications, you can see most websites use it. Services allow you to login using your Twitter or Facebook account for example.

Standards have been created to make it easy to work with centrally managed users. Here are a few that will be interesting for you to look at if you are looking to improve your applications in this area:

• OpenID; a standard for having users log in using an account from one provider to access another provider. In essence it allows users to have one username/password for multiple services. Zend Devzone has a nice tutorial on OpenID to get you started.

• OAuth; OAuth is a protocol designed to handle the actual authentication process across applications. OAuth allows an application to grant access to particular resources to an identity that was provided by another web site. There is a library on google code to make it easier to integrate oauth in your applications.
• SAML; a more enterprise grade way to deal with authenticating users, that includes things such as identify verification and digital signatures so services know that the person that’s logging in is actually the person they think they are dealing with. If you want to work with SAML, you will want to look at simpleSAMLphp, which implements the most important parts of the SAML protocol in a relatively easy to understand PHP wrapper.
• OpenSocial; a standard targeted at social networking, which provides interoperability for users, groups, status updates etc. The OpenSocial PHP client on google code is a good starting point.

For users, groups, privileges and general user data there are plenty of standards to choose from, and all of them have PHP implementations that can help get you started. For other data you may want to develop your own protocols and interfaces, and I hope to have given you enough information to get you started building applications that can get their data from external sources. If more applications are built this way, then the web will become much more pluggable!

This time the contest revolved around creating a class that satisfies a given unit test case.

The test case contained tests for 7 mathematical operations. The contestants did not only have to guess the actual operation, but also provide a class that implements the operation in as few lines as possible.

Two contests for the prize of one

Contest 1 – Smallest solution adhering to decent coding practices

The main contest was just that; because there was some discussion on twitter regarding the ‘fewest lines as possible’ we updated the contest with a rule about ‘good coding practice’ and we ignored coding standard issues; so a brace on the same line or on the next line was considered equally good, and we accepted both regular if constructs and the ternary operator. Some contestants removed all methods from their code and replaced it by a single __call construct, but our jury decided that that is not considered a decent coding practice.

Contest 2 – Smallest possible code size

To cater to those that liked to disregard good coding practices and just create something as small as possible, we had a no-prize side contest for just fame and glory. The entries for this contest are very interesting, we’ll share the top 10 entries we received.

The winners!

As with our previous contests, we have 3 categories. From all the contestants in each category that had all the operations right and had the shortest code we randomly picked a winner. Basically all challenges could be written with one line of actual algorithm code, but you’d be surprised how many needed almost a hundred lines of code to write the class.

Here are our category winners:

• Senior category: David Frame from the United Kingdom

• Medium category: Piotr Młocek from Poland
• Junior category: Luca Lancioni from Italy

Congratulations for winning in your category!

The iPad prize was then randomly raffled amongst the three category winners, and the winner is…..

David Frame. Congrats with your iPad!

The other 2 winners get a ticket to the Dutch PHP Conference 2011.

The solution

Here is a sample class that implements all the algorithms. It’s not a shortest one but it is more clear what the algorithms were in this version:

<?php
class NumberCruncher 
{ 
 
  // Simple addition
  public function operationA($x) 
  {
      return $x+4;	
  }
 
  // Simple multiplication
  public function operationB($x) 
  {
      return $x*3;
  }
 
  // Square / 2
  public function operationC($x) 
  {
      return ($x*$x)/2;
  }
 
  // Factorial (10*9*8*7*6...)
  public function operationD($x) 
  {	
      if ($x <= 1) {   
          return 1; 
      } 
      else {
          return $x * $this->operationD($x - 1);
      };
  }
 
  // Fibonacci (0 1 1 2 3 5 8 13 21 34 55 89)
  public function operationE($x) {
      if ($x<=1) return $x;
      return $this->operationE($x-1)+$this->operationE($x-2);
  }
 
  // Composite of B and C
  public function operationF($x) {
      return $this->operationC($this->operationB($x));
  }
 
  // Composite of D and E
  public function operationG($x) {
  	return $this->operationD($x)-$this->operationE($x); 
  }
 
}

The composite were the most difficult ones and were the ones where most contestants went wrong. operationF was also possible without composite by just providing the simple formula it was made up of, and that’s of course fine as well. operationD and operationE have variants that do not use recursion, but the ones that used recursion were generally shorter (even though it might come at a performance penalty, but that wasn’t a criterium in this contest).

Remarkable

Andy Thompson submitted a nice implementation of operationD. The array_product is a fairly unknown function in PHP, but it’s very suitable for this kind of operation:

  function operationD($d) {
        return array_product(range(1, max(1, $d)));
    }

Another interesting thing to notice is that there are multiple ways to calculate the Fibonacci sequence. Most contestants used a simple recursive formula like the one above. Matthias Steimle was one of the few contestants that used the so-called ‘Golden Ratio‘ (a seemingly magic number that helps calculate the Fibonnaci Sequence and played an important role in Dan Brown’s Da Vinci Code). The golden ratio is the square root of 5, plus 1, divided by 2, and it calculates Fibonnaci numbers in Matthias’ entry like this:

  public function operationE($p) { 
      return round((pow((1 + sqrt(5)) / 2, $p) - pow(1 - (1 + sqrt(5)) / 2, $p)) / sqrt(5)); 
  }

The code minimization contest

It is amazing how small a piece of code can get. We didn’t look for actual solutions to the algorithms; anything /
To make it interesting, let’s walk through the top 10 entries starting with the longest. (Some linebreaks were added by yours truly to prevent horizontal scrolling).

10. Peter Lindqvist (276 bytes)

Here’s the number 10 entry:

<? class NumberCruncher{function __call($n,$a){$p=split(';','+=4;*=3;*=$r/2;=o($r);=f($r);*=$r*
4.5;=o($r)-f($r)');eval('$r=$a[0];$r'.$p[ord($n[9])-65].(is_callable('f')?';':';function 
f($i){return$i>1?f($i-1)+f($i-2):$i;}function o($i){return$i<2?1:$i*o($i-1);}'));return$r;}}

As you can see, minified code becomes quite unreadable, but in Peter’s entry you can still recognize the operations from the algorithm. Peter’s solution is based on using very short function names, a __call interceptor so you only need one function and a bit of eval’d code.

9. Robert Gramm (271 bytes)

<?php class NumberCruncher{function __call($n,$a){$x=$a[0];$s=substr($n,-1);
return$s=='A'?$x+4:($s=='B'?$x*3:($s=='C'?$x*$x/2:($s=='D'?$x<2?1:$x*$this->D($x-
1):($s=='E'?$x<2?$x:$this->E($x-1)+$this->E($x-2):($s=='F'?$x*$x*9/2:
($s=='G'?$this->D($x)-$this->E($x):0))))));}}

Similar techniques, and slightly more efficient.

8. Olivier Garcia (225 bytes)

<?class NumberCruncher{function __call($n,$v){$b=explode(' ','v+4 v*3 v*$v/2 
v>1?$v*$this->D($v-1):1 v>0?ceil(exp($v/2-1)):0 v*$v*4.5 this->D($v)-$this->E($v)');$v=$v[0];
return eval('return$'.$b[ord(substr($n,-1))-65].';');}}

7. Paul Mitchell (214 bytes)

<?class NumberCruncher{function __call($f,$a){$n=$a[0];$d="self::D($n";$e="self::E($n";
$a=split(' ',"*$n*4.5 *0+$d)-$e) +4 *3 *$n/2 ?$n*$d-1):1 <2?$n:$e-2)+$e-1)");
eval("\$r=$n${a[ord(strrev($f))%7]};");return$r;}}

6. Michiel Hakvoort (210 bytes)

<?php
 
class NumberCruncher {
        public function __call($a, $b) {
                $t=debug_backtrace();
                $f=file($t[1]['file']);
                preg_match('#s\(\s*(-?\d+(?:\.\d+)?)#', $f[$t[1]['line']-1], $r);
                return (float)$r[1];
        }
}

This entry deserves bonus points for being smaller than the previous entries, but still remain readable. This is the first entry we encounter in the top 10 that makes use of debug_backtrace to get information from the unittests, where previous entries used to still actually calculate the algorithms. So you can see things getting more dirty from here. :)

5. James Andres (206 bytes)

<?php
eval(gzinflate('=~NÑ
~B@^PE^?Ed@ͤ´~^ÜLJ­·~ BT^L^E~[`w~MTü÷v´z~Z9ç^~F)~ZL^HíÒ>ò~
RG¼Å¢*ùpWSÖOÔÒ´È~ZÆ~D÷^R:k~@C`^@Ú[~CÁ~Q¶~Eg^C*~H&^@\y
b~B~]^[º>9éìc^SÐq-^U%Sä~E~@>^EÉ^\ØÿÝ£ÒI~U~T~H~TXlH~\gAg,ç×
¥@^F «Z0À^@ºëúÆ| ^OD~[^KÉéau~YñR¶^\µò~U5¦þ~E^A| ^_~Yn±qü^@'));

A completely different approach. James zipped his code and used a runtime gzinflate and eval to unzip the code and run it! This makes it possible to pass the unit test using only 206 bytes! Arpad Ray also had an approach where he used zip, but since Arpad entered another entry that was even smaller than his zip version, only James’ zip version made the top 10. (If you have trouble running the above code, I’m pretty sure WordPress will have ruined the character encoding, and I added newlines.)

4. Joris van de Sande (154 bytes)

<?class NumberCruncher{function __call($a,$b){static$i=0;preg_match_all('/\(\s*(.+?)
\s*,/',file_get_contents($_SERVER['argv'][1]),$m);return$m[1][$i++];}}

This is a debatable entry, because Joris assumes there’s a command line argument, which depending on how you run the unit test may not be present. Joris brings us to a range of entries that is more than 25% smaller than the previous one.

3. Devis Lucato (153 bytes)

<?class NumberCruncher{function __call($a,$b){$t=debug_backtrace();$l=file($t[1][file]);
preg_match('#([-.\d]+)\s*,#',$l[$t[1][line]-1],$r);return$r[1];}}

With a gain of just 1 byte over number 4, this code again cleverly uses debug_backtrace and an efficient file/preg_match combination.

2. Tom van Looy (151 bytes)

2 more bytes down!

<?php class NumberCruncher{function __call($f,$p){return 
trim(strtok(end(explode('(',strstr(join(file(__CLASS__.'Test.php')),"$f($p[0])",1))),','));}}

I do have to mention that this one didn’t run out of the box because Tom’s code assumes the test class is in the same directory. If you look at the testcase you’ll see that this is not the case. Since the technique is clear however it’s still a nice entry.

1. Arpad Ray (133 bytes)

And kudos to Arpad for completely squashing the competition by using 18 bytes less than the runner up. As you can see, the winning entry even fits in a tweet! Arpad managed to perfect the simplicity of parsing the testcase using debug_backtrace and a simple split.

<?class NumberCruncher{function __call($m,$a){$b=debug_backtrace();
$f=file($b[1][file]);return+next(split('\(',$f[$b[1][line]-1]));}}

Conclusion

We had great fun running this contest, we hope you enjoyed it too. Keep an eye on http://ibuildings.com/challenge or follow our Twitter account to make sure you don’t miss the next one!

Starting at the Beginning: HTTP

HTTP (HyperText Transport Protocol) is the language of the web, the communication channel over which we send our data. As web developers, we often don’t need to really notice it is there but for web services there are some bits and pieces we should be aware of so let us take a few moments to refresh our memories.

When we surf the Internet, we make a series of requests, and receive responses in return containing the data we asked for. There is nothing special about web services; they work in exactly the same way except that the data in the response is usually marked up for machine-consumption rather than for a browser. When we use a browser, we are aware of the URL that we are requesting and we know that forms send additional POST data. We also observe the HTML response, by viewing the source of the rendered page in the browser (or using cURL, if you’re really hardcore!)

Most web developers are also aware that there are other headers available, controlling cookies, caching and so on, but for web services these form an integral part of the information that is getting sent to and fro between client and server. We observe the headers in one of a number of ways probably the simplest is to use cURL from your command line to observe the traffic. Here’s an example of a request to the google homepage:

$ curl -I http://www.google.co.uk
HTTP/1.1 200 OK
Date: Sat, 29 May 2010 10:39:35 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=375fcb45b134bfca:TM=1275129575:LM=1275129575:S=g68f-bMGCmCJluRj; expires=Mon, 28-May-2012 10:39:35 GMT; path=/; domain=.google.co.uk
Server: gws
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked

Some particular headers are key when working with web services, here is an outline of some useful ones:

Cookie (request) and Set-Cookie (response). These control the cookies that the server gives to the client, and which the client provides on every request. Most PHP sessions work via cookies, and these are handled invisibly by our browsers. When we work with servers, we may have to manually receive, store and provide these when making requests and parsing responses. Many web services are stateless, and so don’t use cookies, but some do use them and it is useful to be aware of these. To be consuming the web from PHP is a bit of a conceptual shift, since usually we’re serving the web, but all the same principles apply.

Status Codes (response). Take a look at the headers shown in the response above: the first line shows “HTTP/1.1 200 OK”. This tells us the version of HTTP that is in use (usually 1.1 although 1.0 does exist and is still used) and also the status of the response. The status code is possibly the most important piece of information in the HTTP response header and this is something that will get a mention again and again in the course of this article. The one shown here, 200, is good news and means that everything operated as expected. The response codes are separated into classes and consist of three digits. Anything starting with a 2 means everything was fine. Something starting with a 3 indicates a redirect of some kind, with a 4 means there was a client error, and with a 5 there was a server error. Some common status codes:

For a full list of status codes, there is a great reference on wikipedia showing various ones (some quite frivolous, in particular look for HTTP 418)

Accept (request) and Content-Type (response). These headers control the kind of content that is included in the response. The client, when making the request, can indicate what format it would like using the Accept header. When responding, the server should set the Content-Type in the response header so that the client will know how to interpret the response.

Having the server include metadata about the response type is really useful and also means that we can use this to check that we got the response we expected. When I run into problems with web services it can often be because in the event of an error, the service returns some default web output as text/html, whereas I have PHP code trying to decode it as XML or JSON … and of course my code then fails horribly because the response format isn’t what I was expecting! Checking the content type of the response before decoding can help to warn us that there has been a problem that we may need to handle.

The Power of HTTP

In the example mentioned above, where checking the content type of a response before decoding can alert us to the problem, we nicely illustrated some of the main advantages of using the functionality given to us by HTTP. If we don’t take the time to set the headers correctly, or we don’t inspect them, we may not be aware that there is a problem or in the worst case our code may just not be able to handle what happened elegantly. Setting these headers correctly in a service, and checking them appropriately when receiving a response, can really help us to write robust services and consumers. If we were to rely purely on information being in the body, as a request parameter or perhaps by decoding the response and then checking for a status code within it, we have lost some of the built-in functionality of HTTP. It is, by design, an envelope protocol to help us understand what is inside without necessarily having to do the work to decode it, or to try to decode it and then fail. Seeing an error status code, a zero content length header, or an unexpected content type will all help us to understand what we have received before we even try to unpack it, which is fantastic!

RPC Services

One of the most common, and in my opinion, easiest service types to relate to is the RPC (Remote Procedure Call). These web services actually feel more like distributed libraries than any kind of new-fangled service-oriented technology … and that is because that is all they are! We make a request to a given endpoint, passing a method name and some arguments, and we get the response back. The easiest way to illustrate what this looks like in PHP is to look at an example. Flickr has a nice RPC service, so let us take a look at an example making a call to their API.

We’re going to call their search method, and look for all photos tagged with “ibuildings”. Flickr has a fairly prescriptive request/response format, so read their docs if you are interested in understanding how this lot goes together. Working with the flickr API is probably a whole article in itself so I’ll try to keep that aspect to a minimum in this post.

Here’s the code, using the PHP curl extension to handle the requests for us.

<?php
// build request format
$xml = simplexml_load_string('<?xml version="1.0" ?><methodCall />');
// method name
$xml->addChild('methodName','flickr.photos.search');
// parameters list
$params = $xml->addChild('params',null)->addChild('param',null)->addChild('value',null)->addChild('struct',null);
// add api key
$child = $params->addChild('member',null);
$child->addChild('name','api_key');
$child->addChild('value','b3ed3314454a875f35d519900c819708');
// add tag to search for
$child = $params->addChild('member',null);
$child->addChild('name','tags');
$child->addChild('value','ibuildings');
 
// make request
$url = 'http://api.flickr.com/services/xmlrpc/';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $xml->asXML());
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
 
$response = curl_exec($ch);
$response_xml = simplexml_load_string($response);
curl_close($ch);

Now if we inspect $response_xml we find a list of photo tags of results that match (if you’re interested, the content of this is the same as this flickr search) and we could use these results to display on another site or as part of an application.

RPC services in general are a nice way to start working with web services – they are a good fit for most applications, and they are easy for us to understand since we already know very well how to work with functions and arguments. They may not be the “coolest” technology around but they are simple and personally I think that counts for a lot! Using RPC services also means that we can easily work with our existing code, in the event that we aren’t always developing shiny new applications from scratch. RPC makes it easy to publish existing libraries as a remotely-accessible service for another system, and also to switch out an existing local library class for a remote one, since the shape of their interfaces is so similar.

The example I showed here with flickr was an XML-RPC service but it is also very common to see and use JSON-RPC and indeed other formats such as serialised PHP arrays/objects are equally valid in this setting. Choosing the right format depends entirely on your application and each has its advantages and disadvantages. Time for a quick tangent to look at these!

Data Formats

When we think of web services, we commonly think of SOAP and other types of XML. In fact these are really useful with web services and are relatively common as a result; we showed an XML example earlier in this post, which handled the XML using PHP’s SimpleXML extension. XML is ideal for communication between machines because it is quite verbose, which means it is precise and leaves little room for interpretation or ambiguity. XML isn’t terribly easy to read without help from machines, which can make debugging a bit harder than working with a simpler format. It is also fairly large in size terms for the data it represents, which isn’t important in most applications but at some edge cases this could make it a less appealing option.

Another very common format in use in web services is JSON. While this format is named after its origins (JavaScript Object Notation), don’t be misled by the title. JSON is in fact easily read and written from almost all programming languages – although for something like JavaScript or an iPhone where relatively limited string handling is available, JSON is an excellent choice. It is lightweight, small for the data it represents and easy to parse although the small size does mean that it is less comprehensive than some of the other formats. In particular JSON doesn’t include information about data types so it can be unclear what format the data was in when it was encoded. Particularly exasperating is that it doesn’t know whether something was an object or an associative array – of course we can work around this but we must be aware of it.

Other possibilities include a serialised PHP format, which is a great choice if you know that client and server will both always run PHP. This format has a lot of the advantages of JSON but does include some type information with the data, which can make it more useful.

SOAP: A Special Case of XML-RPC

With the adoption of PHP rising in the enterprise, it is increasingly common for PHP applications to come into contact with other systems within an organisation written in languages such as Java and .NET. In fact I am seeing more and more situations where PHP is being used as the “glue” to integrate between disparate systems written in these and other “enterprise” languages. In terms of API access, these types of applications will commonly support SOAP through preference, and indeed even within the PHP ecosystem, we see SOAP implementations from various projects aimed more at business than the casual user. In particular, both SugarCRM and Magento provide remote access via SOAP APIs.

Back in PHP 5, a SOAP extension was included in PHP by default, and this makes working with SOAP services, especially those with a WSDL, trivial. As an example, take a look at this example of pulling information from Magento Ecommerce:

    // connect to soap server
    $client = new SoapClient('http://magento.local/api/v2_soap?wsdl=1');
 
    // log in
    $session = $client->login('lorna', 'password');
 
    $list = $client->customerCustomerList($session);

I did say it was trivial! All we need to do is instantiate a SoapClient object, giving the WSDL. We can then call the methods outlined in the WSDL, passing the arguments as we would for a function.

Note: This is an example of the Magento V2 API, for more information about Magento’s API, visit their wiki. This is another topic that would make a whole article by itself.

Strategies for Debugging

As I mentioned earlier, the XML-based formats in general, and SOAP in particular, can be tricky to work with if you need to actually debug them. Part of this is that if your code doesn’t handle SOAP faults or something unexpected goes wrong, it can be really hard to understand what is happening. There is very little more annoying than spending all morning destroying your own code to find the error, only to discover the server has stopped responding correctly! To help with this, I have a few strategies that I would like to share.

First of all, our good friend curl. I worked on one project where I was producing a web service that was consumed by an application my colleagues were building. I told them that I would only accept bug reports against my code if they sent me a replication case which used curl. This was not welcome news to them, but they confessed later that about half of the “bugs” they found turned out to be in their own code when they looked closer. Eliminating the consuming code as a source of problems is really important when working with remote services.

Another key approach, and something I use frequently myself alongside my development, is to use a proxy. Personally I use Wireshark to show me what is happening “on the wire” while I work with web services. I have also heard strong recommendations from industry peers for Charles which is non-free but very inexpensive and seems to be well-loved by its users. Using a proxy, you can check exactly what is getting sent and received without making any alterations to your own code or changing the calls you make, which can very quickly help you diagnose errors.

RESTful Services

Perhaps the purest form of the web services over HTTP is REST. If you are not familiar with RESTful principles then there is a great overview on Wikipedia that explains it really well. Many of the sites publishing information over web services will provide RESTful interfaces and these are easy to understand and simple to consume.

Good examples of these are Flickr who have a great RESTful API alongside all their other offerings, and twitter who use a similar scheme for the URL design throughout their site as well as across their service. This is really interesting, because you can use their services then almost seamlessly alongside their site. Consider this URL:

http://search.twitter.com/search?q=techportal

Opening this in the browser will show the search results for “techportal” on twitter. To request these from PHP in a JSON format is almost as simple, see this example:

$ch = curl_init('http://search.twitter.com/search.json?q=techportal');
 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
 
$json = curl_exec($ch);

We can inspect $json and see that we have the same data in JSON format. The twitter API documentation is rather good so if you’re looking for a good project to practise with, then this could be an ideal place to start!

Web Services

I hope this article has given a good overview of what web services are and how they look when they come in to form part of our code. There is so much more on all of these various topics, and I am sure some of you will want to read more. There are some links for you in the resources section below, and I hope that you will also add your own stories and resources in the comments – thanks for reading!

Resources

flickr API
twitter API
magento API
sugarcrm API
curl – command line http tool
wireshark and charles – debugging proxies

Need to convince your managers about the value of Web Services? Send them over to our excellent whitepaper Introducing Service APIs to explain how services fit in to modern application architecture.

Programmers seem to have a fairly good grasp on ‘how to write code’, but not always what to do with it once it’s been written. This talk with delve into details about how to manage your codebase, and how to manage releases (often intertwined). It will cover topics of coding standards and focus heavily on how you choose to use versioning tools to handle your development and release processes. It will have a distinct Subversion(SVN) flavor to the talk, but the concepts will not be specific to SVN.

In this post, the winners will be announced and we will make some observations based on the contest.

The solution

The shortest route based on the given CSV file is 41799 kilometers and 231 meters, calculated with the Haversine Formula. The route is: Big Ben; Area 51; Zendcon; Golden Gate Bridge; Sydney Opera; Harbour, Tokyo; Mount Everest; Taj Mahal, Eiffeltower; Elephpant Heaven.

The secret CSV file we used can be found here. The shortest route now is 48701 km and 353 meters (Sheffield; php|tek; Mexico City; Apple Cupertino; Vancouver; Hawaii; Chinese Wall; Cape Town; Tower of Pisa; Manneke Pis).

The Winners

Let us have a look at the winners. We had 3 categories; Junior developers, Medium developers and Senior developers. The contest was apparently harder than we anticipated because none of the junior candidates managed to find the correct distance. Two candidates came really close (Elias Sorensen and Matthew Haynes at 42089 and 42090 km on the initial CSV respectively).

In the medium category, there were a few candidates that had the initial CSV right, but only one candidate that also had the correct solution to the secret CSV file. Since we only score on performance, complexity and code size among the candidates with the right solution (see further down for a few observations on this), this person automatically became the winner:

Category Winner Medium: Andy Thompson

Andy’s average performance on 10 landmarks was 333.79 seconds and he used 38 lines of code.

In the senior category, we had many more correct solutions and a variety of algorithms. The top 5 entries with the correct routes:

Among the 2 category winners we raffled an iPad and a ticket to the Dutch PHP Conference. Michiel won the iPad, Andy won the DPC ticket.

Observations

We made a few observations when evaluating the contest and we thought we’d share them.

Adhering to specifications
We automated the test so that we could easily produce results. To do this we had very specific requirements; the script should be called ‘contest.php’, accept 1 command line input parameter indicating the input CSV file, and required the output to be a list of landmark names, one on each line.

It is amusing to see that we received scripts called index.php or ibuildings.php, scripts that had CSV output, and we even received two scripts that given a CSV file of 10 landmarks, would produce a route that would only visit 9 of them. True; elephpants are picky when it comes to choosing places to visit, but the elephpant was required to visit them all. This should not be a problem for an elephpant, especially with landmarks such as Elephpant Heaven in the list; interesting that nobody wondered what Elephpant Heaven is. You will see what we meant when you’re around; it’s not far from the Dutch PHP Conference venue.

Making choices
We noted that Remi was almost twice as fast as Ben, and 5 times faster than Michiel, but complexity and code size were just as important and since Michiel had top score in both categories, he became the overall winner. When I look at the various entries, it’s striking that almost everybody chose to optimize performance as if that were the main goal; I guess it’s more fun to focus on performance. Some candidates used really really fast algorithms (producing results at around 0.24 seconds for both CSV files!), but those algorithms did not guarantee accurate results (ranging from ‘close’ at 42090km to ‘very wrong’ at 76536km. In a real world situation those algorithms might be the best choice if performance is essential and accuracy is not required. In our case however we mentioned that winners must produce the correct result; a requirement like that should be key in selecting an approach to any software problem.

Secrets
The reason we used a secret CSV file to validate the entries was to prevent optimizations that were aware of the route; this proved to be useful since there were indeed contestants that had a very fast correct solution but only because they had knowledge about the landmarks and had for example ‘started their elephpant in a certain direction’. These contestants fell through on the secret CSV file, where they were still performing very fast but not producing the shortest route.

The Traveling Salesman
Most contestants had recognized immediately that this problem is essentially the ‘Traveling Salesman Problem’, which is used in many computer science courses to teach algorithms and optimization. 

We had entries from every country in Europe, still it’s intersting to note that the top 3 were all Dutch candidates. Maybe the Dutch have more traveling salesmen? :)

Running a contest can be hard!
One observation was that we completely underestimated the time it would take to get the results. While some had written very efficient algorithms, others had chosen brute force attacks that ran for quite a while. Since we wanted to get accurate timing results, we ran scripts multiple times and calculated averages. After the first run had run for 3 days and still only had 25% of the entries processed, I changed the script to not run multiple times if the initial result wasn’t under 3 minutes. It eventually still took my test server a few days to go over all the entries.

Although we did not state how many landmarks there would be in the secret CSV file, a larger file could have made the contest virtually impossible. To illustrate how brute force solutions are very difficult to scale: a script that would run for 3 minutes on 10 landmarks could run for 3 hours with just 2 landmarks more.

Another difficulty was the lines of code count and the complexity numbers. We used proper tools to generate the numbers, but the lines of code counter we used would ignore files called .inc and the complexity calculator would ignore code not in a function and so on. So we had to fiddle quite significantly to get accurate numbers there. A lesson for future contests.

Conclusion

For us as organizers, this was a fun challenge, and seeing the buzz on Twitter surrounding the contest it certainly seems the contestants had fun as well.

Let me finish up by saying: Winners: congratulations! And everybody else, thanks for participating, I hope you had fun!

In the meantime the next challenge, dubbed the Test Driven Challenge, has been launched; develop a class that satisfies a given set of testcases. This new challenge will run until June 30 and the prize is once more an iPad. This one is slightly easier than the previous challenge, to give the junior and medium developers another shot at the iPad!

Have fun!