How do we define "the stack"? For the purposes of this article, the stack represents everything between the user and the PHP generating the content they see. As with all performance tuning work, you will need to have a sandbox to try out these techniques, and see what works for you. Often this will be a mixture of several technologies, which I affectionately refer to as the "Speed Sandwich", with each layer doing a specific role to improve the overall performance of the user experience, while aiming to limit complexity to a few distinct components. We want to give the end user the page they requested as fast as possible. So how can we achieve this?

Measuring Stack Performance

Before we even look at what makes up the stack, we need to think about benchmarking. Measuring the effect of your stack on performance is key: indeed, there is no reason to have anything more than the simplest of application servers if we don’t measure performance. Most stacks are a combination of many layers, each with specific tasks and benefits, but these benefits must be weighed against the potential complexity and latency that they themselves add. While TCP communication on the same host is fast, it is not instant (ping is a far simpler communication between machines than HTTP traffic is, and even pinging localhost still takes greater than zero time). This latency adds up (more so if the stack is distributed across many machines), so be sure to measure that the benefits of your stack are still what you believe them to be. There are many measures you can use for determining stack performance: time to first byte (TTFB), first-byte latency, and page load time are good measures to use, but the reality is that the most important measure will be the end user’s experience, so look at your HAR (HTTP ARchive) record in your browser and see if the user experience is being enhanced or degraded by your choices.

Core Principles of the Stack

Before we look at various solutions, we need to identify what we're trying to achieve. Our choices in components will then complement these goals. What are these goals?

1. to serve content fast
2. to have a robust, fault-tolerant solution
3. our users should feel safe and have confidence in their data
4. each component should only deal with the jobs (and traffic) we want them to

With these goals in mind, we can move on to consider the responsibilities we need the components of the stack to cover:

• application server, to serve dynamic pages (in our case, with PHP)
• SSL termination, to protect user data
• static asset serving, for both HTTP and HTTPS traffic
• caching, to decrease both the workload of the application server and the time a user has to wait for a request to be served.

Most modern PHP applications aimed at end users will have all of these responsibilities covered by components within the stack. The question is: which components to use, and how to arrange them in the stack?

SSL Termination

Clearly some responsibilities within the stack are immovable: if the site is going to support HTTPS traffic, then you need SSL termination, and for simplicity, it should be the first layer. When we speak of SSL termination, note that we are talking about the successor, Transport Layer Security (TLS), which most browsers support. SSL is an area with lots of considerations and strategies including asymmetric key size, the algorithm chosen, and CPU of the component responsible, and these factors can really hurt performance if overlooked. To learn more about this area, I strongly recommend the blog of Baptiste Assman, which covers a lot of the general principles as well as how to benchmark performance.

Before we decide on a component to cover this responsibility, we need to consider our architecture as a whole. Essentially there are two common designs: a single gateway such as an SSL load balancer, or distributed nodes as in a cloud architecture using DNS load balancing.

The gateway model, where SSL requests are terminated by a single component, has the advantage of having direct communication with the components behind to establish their state. This means it can subsequently add or subtract them from the pool of nodes available to serve requests. There can be n number of nodes behind this layer awaiting requests, which are now using HTTP, rather than HTTPS, to communicate. There is some overhead between the client and server during SSL handshakes, which can slow down performance. Most components capable of SSL termination have an SSL session cache, which makes subsequent re-negotiations with the client faster if they support TLS. With a single gateway, it is guaranteed that the client will request secure content through the gateway again, making it far more likely that a SSL session (or “ticket” - there is a very good overview of the relevant parlance in this article by Vincent Bernat) can be reused, and thus not require the full handshake again. Note that when looking at fulfilling responsibilities, a component doesn't have to be a separate and distinct piece of software or hardware within the stack: If you've created a private SSL certificate for a local sandbox instance of Apache, then this is essentially the same: a single piece of your stack is solely responsible for all HTTPS requests (in this case, the mod_ssl Apache module).

In the distributed stack, each node handles SSL termination itself, and requests are load balanced by DNS. Each new DNS lookup request is given a different node's IP address, generally by round robin, so that traffic is distributed across the nodes within the pool. This generally scales well: there's no single point of failure as in the gateway model, and it becomes much easier to scale across geographic locations. However, it is less obvious how to remove a failing node out of the pool, so unless rolling out your own DNS service, you are reliant on the implementation of the DNS service provider to define how nodes are added or removed from the pool. Additionally, sessions are “semi-sticky”; most modern browsers will cache the DNS lookup and so return to the previous node, though there is no guarantee that they will. To mitigate the increased latency of creating a new SSL session ticket, there are SSL implementations that support distributing SSL tickets to peers: Stud, the Scalable TLS Unwrapping Daemon, supports passing these tickets to other nodes via UDP, alleviating the overhead of doing this for every single request.

Variants exist between these two models, where there are multiple load balancers responsible for SSL termination for a set pool, and the load balancers themselves are balanced by DNS round robin. Load balancing and SSL termination are not necessarily closely coupled, so the manner of load balancing could be DNS or simple round robin hardware, but at the point you need to know something about the request, you need to terminate. Load balancing with infrastructure without termination does have the disadvantage that sessions cannot be sticky so nodes must either share tickets or the user will suffer performance penalties in handshaking every request.

There are many different implementations for terminating SSL, from a dedicated proxy such as Pound or Stud, to hardware implementations with dedicated accelerator cards, to simply handling this with a Web server. Each solution has its own benefits and drawbacks, so there is no one size fits all implementation. We’ll be looking at the benefits of using SSL for all requests (and the SPDY protocol) in a future article.

Static Assets

Unless the Web application is purely an API, there will undoubtedly be static assets that need to be served to the user. Most common lore immediately says "use a Content Delivery Network!", but before running off to find a CDN provider, consider a few things first:

If your site serves HTTPS traffic, then to gain the full padlock icon within the browser requires ALL assets to be served by HTTPS, otherwise end users may be warned that the page contains insecure items. At the very worst, the site may become unusable if pages rely on JavaScript and the browser blocks access due to being insecure and/or from a separate domain. While there are workarounds, it is a situation that can be avoided if the assets themselves are also available via HTTPS. Many CDN providers support serving assets via HTTPS, but there is wildly differing pricing for this, so keep this in mind when shopping around.

It is always worth considering whether a CDN is needed at all. The rationale of using one is largely to do with providing assets to users in separate geographical locales: massively beneficial if hosting a site in London with a large user base in São Paulo, but less useful if the remotest user using the site is in Watford. There's another excellent techportal article on multi-region deployments which will help you work out if this would help your application.

A simple and expedient method of providing assets under HTTPS, while also gaining the benefits, is simply to have them served by your standard nodes. Many companies have “vanity domains” that simply redirect traffic to the main domain; these vanity domains could be used to provide asset provisioning (so for example, serving assets for foo.org from foo.org.uk), while requests for the base URL can still redirect browsers to the main site. Again, whether or not to shard content onto separate domains is not a straightforward decision (when we discuss SPDY later, this will become clearer), because of the differences between browsers as to how many parallel pipelines they can support.

Another factor to consider with domain sharding is cookies: these are sent only with requests to the parent domain (so in our example, cookies for foo.org would not be sent to foo.org.uk but would be sent to www.foo.org). With some mobile devices only having upload speeds similar to dial-up, considering how large the request itself is can also be as important as the server response, so it may be better to serve assets from a domain where cookies won't be sent and aren't needed.

Caching Dynamic Content

While PHP is sufficiently performant for serving dynamic content, it can become a bottleneck to your users. This is not a weakness in PHP: taking the average Web page, even the fastest super-accelerated C application would be orders of magnitude slower than serving the same content statically. Add in the near-ubiquitous need to communicate with a data persistency, and at high load the application can grind to a halt. There are numerous in-application points that we can cache with a variety of tools such as memcached, Redis or APC, but to use them we've already hit the application layer. Enter reverse proxies.

There are a variety of vendors for reverse proxy solutions, but they all essentially work in the same way: dynamic content sits behind (often described as "upstream of") the proxy, and requests are transparently passed to them. If the response from the upstream server is deemed cacheable by the proxy, then it stores a copy (generally in memory) and subsequent requests receive the cached response directly from the proxy cache.

Not only can a reverse proxy save application latency (in that users get content faster), but load across the application itself will also be reduced. While the majority of queries to a persistence layer are generally read (and so lighter than write operations), these can still overwhelm databases, Redis clusters, and memory. If your application is written in a way that is “friendly” to reverse proxies, then not only are these issues alleviated, but the application can actually be improved.

A well-planned integration with a reverse proxy can allow the application to be brought down briefly without the majority of users being aware that the site is completely being served by the proxy, while more ambitious integrations can use the advanced features of reverse proxies that support Edge Server Includes (ESI), to effectively “widgetize” their site into separate parts, each with individual lifetimes. This allows a page to consist of a mixture of cached and uncached content without having to only provide uncached content.

Tripping Up Reverse Proxies

One of the common pitfalls with reverse proxies is that generally, unless told otherwise, they treat requests with domain cookies appended as uncacheable, regardless of whether the application server needs to use them to produce the response. Generally one of the first things to configure in a reverse proxy are the rules to ignore all but the most essential cookies, to include as a minimum the one used by your application to keep the user's session ID. However, because sessions are a handy place to store transient data, they can be misused. Session misuse means that slow pages cannot be cached and harms the user experience, but in many cases they are absolutely necessary to ensure user Y sees content designated for them and not private content for user X (a common occurrence when first experimenting with caching is seeing a welcome message with the name of the first developer to log in).

Taking an example of a common session implementation, the problem becomes clear:

<?php
// you need to do this to even have the $_SESSION superglobal.
session_start(); // this causes a session cookie to be generated.
if (!empty($_SESSION[‘user_id’])) {
    return true;
}						
return false;

There are a few basic ways to fix this issue:

• Create sessions only when you have to.
• Create sessions lazily, so that they are only created when data is set in them.
• Delete sessions if they contain no data.

Here is an updated snippet that doesn't automatically create a session cookie:

<?php
// is the user logged in?
$sessionCookieName = ini_get(‘session.name’);
// slightly faster to read ini_get() than
// session_name() for some reason...
 
if (!isset($_COOKIE[$sessionCookieName])) {
    return false; // clearly with no session cookie, you can’t have a session ID						
}
// they already have a session cookie,
// so creating one is no overhead.
session_start();
// now check:
if (!empty($_SESSION[‘user_id’])) {
    return true;
}

In this snippet, by checking for the existence of the session cookie, and then only fire up session_start() if it exists, we avoid the side effect of creating a session if we only want to check a variable.

The Application Server

Whatever your personal taste in application server, the core performance increases are in bytecode optimization and output minification. In the Doctrine documentation is a great quote by Stas Malyshev, a PHP core contributor:

“If you care about performance and don’t use a bytecode cache then you don’t really care about performance. Please get one and start using it.”

Whether the bytecode cache is APC, XCache, Zend Opcache in newer versions of PHP, or perhaps the application server is HipHop and the code is precompiled, these solutions can improve performance enormously. Poorly planned include_once and require_once operations can mean a lot of statting the disk, so it is a good practice to try and resolve include paths logically. One approach is to simply run composer with the --optimize-autoloader flag in production, which turns your PSR-0 classloader into a class map. Another might be to write your own autoloader that, after resolving a class, caches the resolved path, or even populates a file of includes to be used by the application.

The final part that will help performance is minification; bytes-on-the-wire savings through taking verbose HTML output and removing whitespace, comments and optional elements can be very constructive, especially for mobile users. Whether this is done as an output filter by the application, the application server, or further downstream is a matter of preference, however, if it does behind a caching layer then the overhead is again reduced, as the minified content will be cached and presented to the user without further optimisation required. We’ll be looking at one possible solution, Google PageSpeed, in the next article.

Laying Out The Stack

After identifying all the responsibilities we want our stack to cover, the temptation is immediately to look at particular solutions to cover these responsibilities, but before we look at those, we need to think about the order of our components.

As already mentioned, some things simply have to happen in a certain order. For our stack it is a fairly obvious sandwich, with SSL termination on the downstream side, and the application server on the upstream side. The choice of whether the layer immediately behind SSL is a Web server or a caching layer depends very much on several factors: if the entirety of static content is served by CDN and all of it is referenced by domain-sharded markup, and you see no need for any post-processing, then there is no advantage in putting the Web server in front of the caching layer. However, if you are looking to do pre- or post-processing, support SPDY, or serve some static elements, then having a Web server in front of the caching layer is a must. A caching layer really should only be concerned with dynamic content, so having the reverse proxy behind a Web server makes sense, as it will only receive requests that by their very nature are for content not servable by the Web server. Additionally, maintaining the caching layer is easier, as the rule sets need not be concerned with static assets at all. Readers familiar with the Varnish reverse proxy may be familiar with statements such as this in the VCL file:

if (req.url ~ "(?i)\.(jpeg|jpg|png|gif|ico|swf|js|css|gz|rar|txt|bzip|pdf)(\?.*|)$") 
{
    // ignore because this is a static file....
    return(pass);
}

Blocks like these are no longer needed if the static assets are served in front of the cache and only dynamic content can reach the caching layer. Here’s a snippet of an nginx configuration where PHP pages or content not in the docroot are passed upstream:

http {
...
	# Address of Varnish server
    upstream _varnish {
        server 127.0.0.1:6081;
    }
}

server {
...

    location / {
        # This is cool because no php is touched for static content.
        try_files $uri @backend;
    }

    # pass all PHP requests upstream
    location ~* \.php$ {
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        proxy_pass http://_varnish;
    }

    # Clearly it doesn’t exist in the docroot, so maybe it’s dynamic.
    # Let’s pass this along to the caching layer
    location @backend {
        proxy_pass http://_varnish;
    }
}

With the above Web server in front of the caching layer, you can be fairly certain that the traffic reaching the reverse proxy is for dynamic content. Additionally, we can do some interesting things with post-processing to preserve caching, and this is a topic we'll return to as part of this article series.

What’s next?

Hopefully by now you have a clearer picture in your head of the responsibilities that need to be covered by the application stack, and an idea of where in the user request-response flow these need to be addressed. We asked earlier what components we should to address these; the answer to that is very much to do with your own needs and preferences. So in the next group of articles about performance, we’ll look through some possibilities for the application layer, caching layer, and Web layer, starting with Apache and mod_pagespeed.

                          

Now you're convinced that automated testing is important, but isn't unit testing enough? Unit tests are cheap, fast and small. Why might you want to expand into using a technique such as functional testing? Once again the answer is confidence. The more complicated an application becomes, the more complicated the interactions between separate parts of the application become. Just as you can't be confident your car is roadworthy by manually turning each of its wheels independently, you can't be sure that an entire application is working by testing each of its units independently. You need functional testing for complex applications - and today, that's every application.

Silex

Silex is a PHP microframework based on Symfony2 components. It has rapidly become a popular tool in the Symfony2 community thanks to its simplicity and shallow learning curve. That said, the simplicity of Silex does not mean that it is only fit for simple applications, in fact the opposite is true. Some types of application, like RIAs (Rich Internet Applications) are better built with PHP and the minimalism of Silex.

As an application becomes more complex, so it becomes more important to properly test it. Silex provides some tools for functional testing using the Symfony2 BrowserKit component. This is useful, but what if some of of the application's functionality requires real HTTP requests to be made? Those could be done using a console browser emulator like Goutte. How about if some functions depend on AJAX to behave properly? Using Selenium or even Zombie.js could be a good option in that case.

There are two problems with all the solutions mentioned here:

1. There's simply no single best choice in terms of browser emulation. Some emulators (BrowserKit, Goutte) are extremely fast, but don't support AJAX. Others (Zombie.js, Selenium) support AJAX but are much slower. In an ideal world, you would want to use multiple emulators at the same time, choosing the best one for each specific case. That leads us to problem #2.
2. Different browser emulators are written in different languages (PHP, JS, Java), using different libraries and provide extremely inconsistent APIs for developers to work with. That means it is difficult to switch from one emulator to another as it will require all helpers and testing tools be rewritten.

Mink

To help solve some of these problems, a library called Mink was written. It was created originally as a complementary tool for Behat, but is not tightly coupled to Behat itself and so it can also be used with any framework or tool out there. Mink is a browser emulator abstraction layer - a tool that removes the differences between the various browser emulators and provides one single API through which you can control them all. This essentially means you can switch from browser emulator to browser emulator quite seamlessly as they all use the same PHP API. This article will show you how to use Mink to functionally test an example Silex application using a couple of different browser emulators.

Setting Up

As our test application we'll use a very simple Silex blog application that is already prepared for you. The code is on github at https://github.com/everzet/silex-mink so use that as your starting point.

Set Up the Test Suite

In order to install and integrate Mink, Silex and PHPUnit, some extra entries are needed in composer.json. The new section should look something like this:

"require-dev": {
    "phpunit/phpunit":               "3.7.19",
    "behat/mink":                    "~1.5",
    "behat/mink-browserkit-driver":  "~1.1",
    "behat/mink-selenium2-driver":   "~1.1"
},

These changes ask Composer to install the latest versions of PHPUnit, Mink and couple of Mink drivers in the "dev" environment. To effect these changes, run the following command:

$> php composer.phar update --dev --prefer-dist

Drivers are what gives Mink knowledge about different browser emulators (Symfony2 BrowserKit and Selenium, in this case). Mink supports quite a selection of drivers out of the box:

• BrowserKit - Symfony2 TestClient driver. Extremely fast headless emulator without isolation or AJAX support.
• Goutte - written in PHP browser emulator. Fast fully isolated headless emulator that doesn't have AJAX support.
• Sahi - Sahi.js driver. Extremely slow real browser controller. Has best AJAX support in town.
• Selenium2 - Selenium v2 driver. Slow real browser controller. Has decent level of AJAX support.
• Zombie.js - Zombie.js driver. Fast headless browser emulator with AJAX support. Known to be buggy in some cases.

For demonstration purposes, we'll install only the BrowserKit and Selenium2 drivers, however the technique to switch between them is the same for switching to any other Mink driver.

PHPUnit setup

Next step is to create our PHPUnit test configuration. Put inside your phpunit.xml:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit backupGlobals="false"
         backupStaticAttributes="false"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="false"
         stopOnFailure="false"
         syntaxCheck="false"
         bootstrap="vendor/autoload.php"
>
    <php>
        <server name="BASE_URL" value="LOCAL_APP_URL" />
    </php>
    <testsuites>
        <testsuite name="Silex app test suite">
            <directory>./tests/</directory>
        </testsuite>
    </testsuites>
    <filter>
        <whitelist>
            <directory>./src/</directory>
        </whitelist>
    </filter>
</phpunit>

Nothing really special here, except maybe the BASE_URL server variable. We'll use this in our test cases as this is a test suite environment detail.

Base Mink Test Case

Now we're ready to start writing the first Mink test case for the Silex app. First let's create a base MinkTestCase class, which can be extended by each of the functional test classes. I put the following code into tests/functional/MinkTestCase.php:

<?php
use Silex\WebTestCase;
use Symfony\Component\HttpKernel\Client;
use Behat\Mink\Mink;
use Behat\Mink\Session;
use Behat\Mink\Driver;
 
abstract class MinkTestCase extends \PHPUnit_Framework_TestCase
{
    private static $mink;
 
    public static function setUpBeforeClass()
    {
        if (null === self::$mink) {
            $app = require_once('../PATH_TO_YOUR/app.php');
            $app['debug'] = true;
            $app['session.test'] = true;
            $app['exception_handler']->disable();
 
            self::$mink = new Mink(array(
                'silex'    => new Session(new Driver\BrowserKitDriver(new Client($app))),
                'selenium' => new Session(new Driver\Selenium2Driver),
            ));
            self::$mink->setDefaultSessionName('silex');
        }
    }
 
    protected function teardown()
    {
        $this->getMink()->resetSessions();
    }
 
    protected function getMink()
    {
        return self::$mink;
    }
 
    protected function getSession($name = null)
    {
        return $this->getMink()->getSession($name);
    }
 
    protected function getPage($name = null)
    {
        return $this->getSession($name)->getPage();
    }
 
    protected function assertSession($name = null)
    {
        return $this->getMink()->assertSession($name);
    }
}

The main area of interest at this point is the setUpBeforeClass() method. This method sets up Mink with two preconfigured browser sessions (one Silex, one Selenium) backed by BrowserKit and Selenium2 drivers. We also set a Silex session as the default, which means that when no $name argument is provided to the helper methods, they will work with this exact session.

The other 3 methods are helper methods which will be very useful when writing test cases:

• getSession([$name]) returns a Mink session object.
• getPage([$name]) returns a session webpage object.
• assertSession([$name]) returns a session assertion object.

Writing the First Test Case

With a preconfigured base TestCase class, the first test can now be written. Let's test that we can see and fill in the article form. In my project I added this to tests/functional/ArticleTest.php:

<?php
 
require_once 'MinkTestCase.php';
 
class ArticleTest extends MinkTestCase
{
    public function testThatArticleFormIsAccessible()
    {
        $this->getSession()->visit($_SERVER['BASE_URL'].'/');
 
        $this->getPage()->clickLink('Add article');
        $articleForm = $this->getPage()->find('css', 'form');
        $this->assertSession()->fieldExists('Title', $articleForm);
        $this->assertSession()->fieldExists('Body', $articleForm);
    }
}

Let's work through the code snippet above:

1. Visit the application homepage using the Session::visit() method and the BASE_URL.
2. Find the article form using CSS identifiers, and assign the resulting DOM object to a variable called $articleForm.
3. Make an assertion that a field with an attribute 'id', 'value', 'label' or 'name' containing 'Title' exists inside the form stored in $articleForm
4. Do the same for a 'Body' field.

Now invoke PHPUnit to run the new testcase:

$> vendor/bin/phpunit

The test completes quickly; this is because by default it runs against the Silex browser session, which uses the BrowserKit driver, which in turn uses the internal Silex TestClient.

Were you expecting something more complicated? Let's add another test case, but this time filling in the fields on the form and checking that a preview area is shown:

public function testArticlePreview()
{
    $this->getSession()->visit($_SERVER['BASE_URL'].'/add-article');
 
    $this->getPage()->fillField('Title', 'Hello Mink!');
    $this->getPage()->fillField('Body', 'Func testing is fun!');
    $this->getPage()->pressButton('Preview');
 
    $this->assertSession()->elementTextContains('css', '.preview', 'Hello Mink!');
    $this->assertSession()->elementTextContains('css', '.preview', 'Func testing is fun!');
}

Again, let's work through the example in order:

1. Visit the /add-article page.
2. Fill the 'Title' field with a value of 'Hello Mink!'.
3. Fill the 'Body' field with 'Func testing is fun!'.
4. Then press the 'Preview' button.
5. Assert that element identified by a specific CSS selector contains some text.

Run the entire test suite again:

$> vendor/bin/phpunit

For completeness we should also test that preview area is hidden by default, which is fairly simple:

public function testPreviewIsHiddenByDefault()
{
    $this->getSession()->visit($_SERVER['BASE_URL'].'/add-article');

    $this->assertSession()->elementNotExists('css', '.preview');
}

Hopefully the examples have seemed quite straightforward so far. Perhaps you are wondering though whether this could have been done with a default Silex function test case?

In fact, we can't quite do it with the other tools. One major benefit of Mink is a much cleaner API, which is consistent across all browser emulators. This potentially means that both our tests could be checked against different browsers with minimum effort.

Remember that optional $name argument to the helper methods? By providing "selenium" as an argument, it is possible to force Mink to use the Selenium session and Selenium2 driver for each written instruction. Multiple sessions and drivers can be mixed inside a single test case. You can even have multiple sessions using same driver working together to test, for example, a chat application.

For demonstration purposes let's switch all of our test cases to use Selenium2. How? By changing the default session name. Add this to your test class:

protected function setUp()
{
    $this->getMink()->setDefaultSessionName('selenium');
}

At this point, you should download and run the Selenium Server JAR from the Selenium website and then run the test suite again:

$> vendor/bin/phpunit

When you run the tests this time, an actual browser (firefox by default) should have opened and performed all the actions in the tests. That's the power of Mink. It works exactly the same way for any Mink driver, such as Zombie.js or Sahi.js.

Are you using Mink? Would you like to? Tell us what you think in the comments - and happy testing!

Additional Resources

• You can read more about automated testing on wikipedia.
• Information about Mink can be found on official website.
• There is documentation for all available Mink methods on the Mink API website.

                          

Day One

The gathering started in front of the Scottish parliament where a crowd of about 100 people slowly grew. We then got on coaches to Airth castle, how many other conferences can get credited with being held in a Scottish castle? After some refreshment and the registration process, we got straight into the mood of the conference.

How Google Builds Web Services (Ian Barber)

Ian's talk was quite technical and very interesting; it gave fresh perspective on how web services and APIs should be designed and the importance of isolating your core API from third parties. If you have an API you are proud of and you want to share it, consider creating a wrapper for it rather than exposing it as it is. Ian showed how Google failed to avoid coupling the core API feature set to clients requirements in the past, by discussing how products like GData didn’t have the expected success because of lack of consistency, documentation and a common interface.

He then demonstrated how Google are now providing a new API layer on the top of all their services, how it’s been documented, and how everyone can now easily access it via a "discovery file" that describes the APIs.

Building Better Developers (Rowan Merewood)

Rowan introduced the crowd to various practices we constantly use here at Inviqa, and he also gave an idea about what we should look for as a developer in order to have fun everyday and keep on enjoying what we do. It’s not the amount of things you know that makes you better but it’s how efficiently you do what you are good at; to improve that, a little bit like an athlete you have to practice, practice and practice. It’s important to realise you work in a team, whether this team is within your company or within the community, so it is better to collaborate than to be a superhero.

Real Time Web Apps in the Wild (Phil Leggetter)

I was impressed by the number of use cases and applications running on Pusher and how we can nowadays make the web more immediate where the technology allows us, however I would have expected less of a marketing focus to the talk. I'd have preferred to see a showcase about how to easily implement the Dolphin race game that was demonstrated; this would have been more relevant to the developer audience.

Identity, responsiveness and the future of the web (Ade Oshineye)

The last talk of my day and probably one of my favourites of the entire event. Ade is an incredible speaker and I was excited to hear him tell how all these responsive design buzz words are leading us in the wrong direction. He clearly pointed out how developers should stop focusing on the idea of content, while they should instead pay more attention to the context. While surely content is relevant and important, Ade asked the crowd what a mobile user wants? And the answer is simple: he wants features, features which are beneficial to him having a better service and experience from the service provider.

Evening Festivities

After Ade’s talk, in perfect Whisky Web tradition, we then all moved into Airth Castle itself for our whisky tasting masterclass - this year sponsored by Dewar’s. Sadly for me their production specialises on blended whisky while I happen to be an Islay single malt lover, however they were able to impress me with their full cask strength single malt used as base of any of their blends.

Day Two

Day two took us back to the centre of lovely Edinburgh. The event was held within the walls of the beautiful Surgeon’s Hall, the major medical museum in Scotland, considered a museum of national relevance and one of the many tourist attractions in Edinburgh.

Coming Soon to a Browser Near You (Alan Greenblatt)

Do you remember the days when Adobe was leading the market when people were talking about interactive online experiences? Things have changed quite a bit since then, but it seems like this amazing company is coming back with some interesting ideas. No more flash this time, simply a pioneering approach to HTML5 and other delights that the W3C is planning for us in the near future. WebSockets is just the beginning of the revolution and Adobe seems to be willing to push the standards; bringing tools to meet our publishing needs into our browsers, offering the ability to shape content with ease and present our website like a press magazine in a way that has been impossible until now.

The down side? These experimental features only work in the latest chrome dev build and not even automatically. Let’s see what will happen in the future!

Scaling Applications with RabbitMQ (Alvaro Videla)

If your application needs a focus on performance, scalability, or language-agnostic communication between services, then a message queue system could be your answer - and RabbitMQ possibly one of the best. What Alvaro highlighted very well during his talk was that having a good messaging queue service in place will help you improve decoupling between your systems.

Decoupling between systems means better load distribution, better control on your streams of information, and also an improved user experience. For example in a Publisher/Subscriber scenario, the application does not need wait for something to happen since the relevant message will be provided at the appropriate time, allowing me to carry on doing other things without any need to worry.

3rd Talk: Open Search (Chaitanya Mishra)

Before I saw this talk, I was aware of OpenGraph by Facebook, but I didn’t know that Facebook plan to release a new feature called Open Search on top of it. Open Search will basically allow people to perform complex queries on any public content stored on Facebook servers using a human-readable language. A simple example, we will be able to type a phrase like “photos of people living in London who are friends of my friends” and this will produce the expected results. The Facebook engineer gave a complete overview of the project, its current status, how they architected the system, and how they solved some of the performance problems they encountered while developing this incredible and amazing feature.

Closing Keynote (Josh Holmes)

Josh did a great job last year opening the conference, and this year he did another great job closing the ocnference talking about change. In a different light he once again tried to motivate people to do more, to search for innovation, to experiment, to share ideas, to share credits and fame, as well as failure. At the same time he tried to induct responsibility, and probably the most memorable sentence I took with me out of his talk was “Don’t change just for the sake of it”.

Wrap Up

Whisky Web’s organisers wanted to create something different once more and once more they succeeded in their aims. This year they created a smaller conference with a limited number of available spots to give people a sense of familiarity and to enforce human interactions. One of their goals was “we want you to leave this conference having spoken with everyone” and I think that all the attendees created new connections and strengthened old ones. Many thanks to everyone who made this event happen.

                          

This talk takes the listener through the process of designing an API structure up front that uses hypermedia at it's core over HTTP, what considerations do you need when selecting a media type to represent your resource and what is out there to help you document it for others to use. This session is recommended for architects and developers alike and will give a good grounding in writing excellent, self-explanatory Hypermedia APIs.

The video below is a recording of Ben's conference talk at PHPUK 2013:

Video Courtesy of the PHP UK Conference organised by PHP London

Ben is technical assurance manager at Inviqa (Sheffield). He has many years of professional experience in a variety of programming languages including C, C++, Perl, Python and has been working with PHP for the last 6 years. Throughout his career he has worked at a variety of places in different industries; mobile marketing, data, ISP and in professional services all of which has given him diverse experience of software engineering in different fields.

Outside of work he is a father, and with what little spare time he has left he can be found hacking one of his own projects (lots of ideas, not enough time!), playing on his PS3, or infrequently blogging on techPortal and his own personal blog at http://nocarrier.co.uk

If you'd like to learn more about the Hypermedia API and creating RESTful applications Ben has devised a training course which gives an in-depth understanding of REST and how to implement it. It covers HTTP, Hypermedia, HATOAS, SOA, and more advanced topics like testability and intermediaries. Creating RESTful Hypermedia Applications

                          

In this talk Marcello explores efficiency in contrast to effectiveness. He looks into how practices, traditionally accepted as efficient, sometimes turn out to be less effective than a few "impractical" things he has come across.

The video below is a recording of Marcello's popular conference talk at PHPUK 2013:

Video Courtesy of the PHP UK Conference organised by PHP London

Marcello Duarte is the lead developer of PHPSpec and heads the Training at Inviqa. As an agile coach, he's coached many teams in many organisations into Agile processes and development practices like Scrum, TDD, Refactoring, Collaborative Product Ownership and Software Quality.

                          

What happens though if you complete your analysis and you do have a bottleneck, and it’s going to take a while to fix? Or you have some sluggish legacy code that is a huge lump of technical debt? Fixing sites is hard, and as engineers, we are frequently attracted to the most difficult technical challenges. We want to fix the problem, not mask it. That said, there are many ways we can deal with performance without actually fixing the problem, while keeping users even happier.

Speed vs Performance

First let’s quantify what we’re talking about here: speed. What is the difference between speed and performance, you may ask? The core of it is this: you care about performance, but your users care about speed. The average consumer of your site does not care how fast or slow your service layer is, nor about the efficiency of your database calls. They simply want to use your site as fast as possible, and an ever-increasing number of them are accessing your site on mobile devices.

Speed is therefore more than just how fast the page loads, but also how fast the user perceives your site to be. So, how do we improve this? Well, there is no single fix that will make your site incredibly fast, although some will have more or less effect than others. It requires a holistic approach of a great many small modifications.

David Brailsford is the Performance Director of the British Cycling team - a team that has come to dominate world cycling, with 65 medals at the London 2012 Olympics, 29 of which were gold. In an interview after the Games, he cited, in addition to crediting the athletes themselves, part of their success to Matt Parker, Head of “Marginal Gains”:

"The whole principle came from the idea that if you broke down everything you could think of that goes into riding a bike, and then improved it by 1%, you will get a significant increase when you put them all together"

(full article: http://www.bbc.co.uk/sport/0/olympics/19174302)

This methodology works very well with websites too. What are these marginal gains we can make? They fall broadly into three categories:

1. Give the user what they want (and only what they need).
2. Give them what they need faster.
3. Don’t make them ask for it lots of different times if you can give it to them all at the same time.

All of this seems fairly straightforward, and some of this has already been covered in the post on web performance tips on the main Inviqa blog. There is a another collection of tweaks however, which I guess could be described as “magic” - or more accurately, illusion.

These “sleight of hand” tricks give the illusion of a fast site, even if something is taking a while to do. Recently a friend asked me to put together a website for his cottage industry. He paints miniatures for tabletop wargamers, and he has a level of control with a brush that is simply incredible; I could easily believe he could put makeup on an ant. As you can imagine, his website is very much about showing off his work, and so he has lots of high resolution images of models he has painted. These images are stored on Flickr and displayed via a jQuery carousel on his front page.

The problem is of course, that these images are huge, and his customers want to see his handiwork right away. You can optimize, crop, and play with the image only so much before a site showing off artwork is soon showing off fuzzy quality images that detract from the message he’s trying to convey. In the end, we put up his favourite image from the set statically on the page, then as soon as enough images were loaded, replaced the image with the carousel, fading in the new content. The effect was that his users believed that the page was loaded extremely quickly, as opposed to waiting for the carousel which would have led to the page taking many seconds to load.

Of course, this isn’t limited to static binary assets for giving the user an illusion of speed. A popular website design is to have a navigation bar across the top of the page, indicating the user is logged in by their username and/or avatar being present. If a user wishes to modify their profile settings, this will mean fetching data from one or more sources to populate the forms. But what do we already have? Well, we can at least populate part of the form with the same details we have present in the navigation already, and then bring in additional details via AJAX. The net result is that the site feels more active to the user, because they have something they can do immediately.

Don’t Overlook Mobile Connectivity

These days, when Web developers refer to “responsive” websites, we are referring to Responsive Web Design. “Responsive” can also mean reacting or replying quickly or favourably; this becomes more important to consider in design as users of websites increasingly interact via mobile devices. We tend to forget while working on a laptop with a constant Ethernet or reliable Wifi connection that this is not the user experience of someone out and about on an iPhone.

Emulating the experience of a mobile user is very hard to emulate if your development environment is a VirtualBox instance running on your local machine. Network latency can of course be synthesized (check out our previous post about Charles), but it isn’t a natural step, and you would want to switch it off when not testing for front end performance. The other downside is that it can give you a marked advantage to your local instance versus assets on a CDN (Content Delivery Network): given a round-trip to get jQuery from your local /js directory, compared to getting it from something like the Google Ajax APIs, Google will seem slower, which is unlikely in reality. As a result of all these factors, I recommend that you get a “playground” - fire up a cloud instance and test performance against that.

Are Public CDNs Right For You?

Since I just mentioned the Google Ajax libraries, this is a good time to talk about specifics. A lot of developers immediately assume that if they use Google to provide a library such as jQuery, then you have improved your site loading times. This is possible, and there is also the advantage that Google serves these under both HTTP and HTTPS, allowing you to use the non-scheme-specific URL //ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js to get the file. If your site only used this file for JavaScript, all is well, but if not then there is a "gotcha". Let’s take a popular example of an HTML5 boilerplate, initializr. This gives you a responsive layout, Twitter Bootstrap, jQuery and Modernizr, all in one bundle. It’s one of my personal favourites, so let’s look at some (abridged) code:

<!DOCTYPE html>
<!--[if lt IE 7]>      <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]>         <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]>         <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
    <head>
        <meta charset="utf-8">
	   …More head tags....
        <script src="js/vendor/modernizr-2.6.2-respond-1.1.0.min.js"></script>
    </head>
   <body>
       …The main body, then...
        </div> <!-- /container -->
        <script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
        <script>window.jQuery || document.write('<script src="js/vendor/jquery-1.9.1.min.js"><\/script>')</script>
        <script src="js/vendor/bootstrap.min.js"></script>
        <script src="js/plugins.js"></script>
        <script src="js/main.js"></script>
...
</body>

This works well, but you have five JavaScript files being included on this, and probably every other, page:

• modernizr-2.6.2-respond-1.1.0.min.js

• jquery.min.js from Google’s CDN
• bootstrap.min.js
• js/plugins.js
• js/main.js

As these are across two domains, that’s two DNS lookups and six requests (one for the page itself, five for the JavaScript alone, and we haven’t touched CSS yet). Compare this with creating an init.js file with all of these files combined, it becomes one DNS lookup (which the browser has already done, to find the page) and two requests (again, including the page). In such a circumstance, is it worthwhile using the Google CDN? You could of course combine just the local files together, resulting in two DNS lookups and three file requests, but you need to actually play around to see which works for you (read Adam's post on multiregion deployments for examples on how to establish what works best).

There's a common misconception that, by using jQuery directly from Google, there is a greater chance of it being already cached in the browser from visiting a previous site, and that the browser will have already done the DNS lookup. However, as most sites want to link against a particular version of jQuery for stability, this can mean dozens of minor versions. Additionally, Mobile Safari doesn’t persist cache over application restart, so the chances are smaller than you might initially assume. Relying on your user already having done something elsewhere is not a strategy to rely on for performance, so always test your assumptions in your playground.

Prefetching Domains

While browsers are getting better at preprocessing HTML to gather domains referenced in the HTML ahead of schedule, not all of them are, and there also cases where content is dynamically loaded in from another domain. In the earlier example, an API feed from Flickr was consumed, and then the images were displayed directly into the carousel. When the browser comes across this API feed (and subsequently the images, as they are static files and only the information on how to concoct their URL comes from api.flickr.com) it has to do a DNS lookup, adding typically 10-30ms to each request. Taking the worst case, that’s 30ms to get the API feed, and another 30ms to get the IP of whichever static farm the image is hosted on! Adding 60ms to the lookup time is something that can be avoided, however. Let a browser know that somewhere on this page you will need content from a particular domain by using dns-prefetch tags just after the opening <head> tag:

<link rel=dns-prefetch href="//ajax.googleapis.com"/>

This tells browsers that support dns-prefetching that they should, in parallel, look up the domain specified because it will be needed at some point. This prevents DNS blocking when fetching assets. There is some debate as to whether domain sharding is still a good technique, but if you use it, then prefetching will also speed up browsers if the asset domains are completely separate. The issue is muddied by more recent browsers ignoring the RFCs on HTTP pipelining, so again, test your assumptions.

Using JavaScript Passively

So far the DNS lookup times have been improved, one large file has been created that has all of the site-generic JavaScript in it. Two questions then:

• how do we manage this file and dependencies
• how do we use it?

It is common practice to place the JavaScript includes right before the closing </body> tag, or at least add defer or HTML5’s async attributes:

<script async src="foo.js"></script>

However, we want to also put JavaScript throughout the body of our page, and the techniques covered so far don’t deal with dependencies, only with loading the library in a non-blocking way. If we think of a simple Bootstrap page with three columns, and we want the leftmost column to contain the latest Tweets from our Twitter account, then the JavaScript to initialise this is best placed within the block. As we widgetize our sites more and more, we want to write small, easily-reusable components, and that means that we have JavaScript within the same logical block as the element that it relates to. If this application uses a templating engine that is aware of placeholders, we could append this block to the bottom of the page, but we still have a dependency: the library we are relying on must be present.

Most of the time we work on the premise that if we put a library like jQuery in the <head>, it is hopefully there by the time we use $(document).ready(). In browsers that block while loading JavaScript in the <head>, this is guaranteed, but in that case, any further page processing is blocked until the file is available. More and more browsers are behaving better when getting elements referenced in the head of HTML, so it is increasingly likely that the library is there when it is needed. However, with 3G mobile devices having an average connection speed of under 2Mb, this is never a certainty, especially if your first use of it is early in the body of the page.

We need to defer until at least the core dependencies are in place. Let’s take a look at some code, modifying again our initializr template.

<!doctype html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7" lang="en" xmlns:fb="http://ogp.me/ns/fb#"> <![endif]-->
<!--[if IE 7]>    <html class="no-js lt-ie9 lt-ie8" lang="en"> <![endif]-->
<!--[if IE 8]>    <html class="no-js lt-ie9" lang="en"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en"> <!--<![endif]-->
    <head>
        <meta charset="utf-8" />
        <link rel=dns-prefetch href="//ajax.googleapis.com"/>
        <script>
        	var tp = window.tp || {};
        	tp.fn = tp.fn || new Array();
    </script>
    <!-- rest of head from initializr-->
    ...
    </head>

What is happening here? The script is creating a small JavaScript object called tp (for TechPortal), and defensively checking to see if it already exists, in case this script is being included in the page in an unexpected way (such as Lightbox, or an Ajax call). We also give it a property, fn, which is an array to hold callbacks.

Now let's add a JavaScript call to the page. The earlier example was a Twitter feed, so let’s assume we have a file, tweets.js, which requires the jQuery UI library. I’m basing this example on the Sea of Clouds Twitter widget, with some additional jQuery UI functionality thrown in. Our template requires Modernizr to work properly, so we’ll simply use the built-in Modernizr loader, which is based on yepnope:

<div id=”tweets” class=”span3”>
    <script>
    tp.fn.push(function() {
        Modernizr.load([{
      load :[
            '//ajax.googleapis.com/ajax/libs/jqueryui/1.10.1/jquery-ui.min.js',
      '/js/vendor/tweet/tweet.min.js'
      ],
      complete: function() {
          $("#tweets").tweet({
              join_text: "auto",
              username: "techportal",
              avatar_size: 48,
              count: 5,
              auto_join_text_default: "",
                auto_join_text_ed: "",
                auto_join_text_ing: "",
                auto_join_text_reply: "",
                auto_join_text_url: "",
                loading_text: "loading tweets...",
                refresh_interval: 0
            });
        }
    }]);
});
</script>
</div>

Here, an anonymous function is appended to the fn array, and Modernizr.load() is used to retrieve the dependencies. Once these have been loaded, the tweets are then initialised, targeting the named <div>. However, this code won’t actually do anything just yet, because it is simply a function, and nothing has called it.

Let’s create a file called init.js. We’re going to use this file on every single page of the site, so we’ll prepend minified jQuery and Modernizr to it to reduce requests:

// init.js
 
// Modernizr and jQuery go here...
$(document).ready(function() {
	// we know that jQuery is loaded as it is prepended, and now we know
	// that everything else is ready to go too.
if (window.tp) {
for (i in tp.fn) {
	// iterate through all the stored functions...
      		var fn = tp.fn[i];
		// and then execute them....
            fn();
    	}
}
});

And now we append this to our example page:

<!-- cover all bases and use both async and defer attributes →
<script src=”/js/init.js” async defer></script>
</body>
</html>

Using this method, we ensure that there are no blocking or dependency issues for the JavaScript on page load, while also allowing us to have libraries available without having to have advance knowledge of every single widget on a page. The page will load without needing any JavaScript until it reaches the closing </body> tag, then call in the site-wide Modernizr and jQuery libraries in one file, reducing requests and DNS lookups. After that, it will iterate through all the page-specific JavaScript and sort out any dependencies as necessary, and finally execute any additional functionality nested within these latent functions.

We could also add into init.js all of our bootstrap and responsive files as well, although if there is a lot of HTML on the page, it may take a while before the browser reaches the tag to include it, which could leave the site looking odd for a second or two until the responsive elements are brought in. In which case, experiment with placing it in the <head>, keeping the defer and async attributes, and see how performance is affected by this change.

As the loop to call the anonymous functions is wrapped inside jQuery’s $(document).ready() event handler, it should still gracefully wait until the page is loaded.

Test, Test, and Test Again

You should (with effective analytics and logging) have an idea which browsers and devices are being used by your users, and be able to test to target these. Other considerations could include whether tweets.js is the only library that needs jQueryUI; if so, is there a benefit to having the two libraries separate, or instead combining them into one file with jQueryUI prepended? If there are only a few libraries like this, all using jQueryUI, another option is having a libraries.js file where all the libraries live, or including jQueryUI in the site-wide init.js, or simply having all of the JavaScript inside init.js. Again, test to see what works for you, and more importantly, your users.

This also applies to CSS files as well: you need to find what works for a specific application. A really handy tool for concatenating and managing CSS, LESS and JavaScript is CodeKit. It watches your project folder and then automatically generates aggregated files. With it, I can easily choose to prepend or append libraries, process LESS files and output CSS, and also choose to minify the output if I wish. This means I don’t have to worry about writing my custom JavaScript and CSS files in a minified way: I simply have the files in a resources directory, and tell CodeKit to output the files into the /js and /css folders in my html directory. It also has the YUI Compressor and JSLint built-in, allowing you to produce very small aggregate files and also avoid obvious parsing issues in your aggregated JavaScript libraries. CodeKit isn’t free, but I find the licence fee worth it for the time it saves me. Alternatively, you can use node.js and have the compilation of your LESS files and JavaScript validation as part of your build process.

In my next article, I’ll be looking beyond optimising the HTML and static assets of the page and look at how we can optimise the stack to improve delivery of your pages.

                          

I arrived in plenty of time to enjoy the complimentary bacon sandwich and coffee on offer before finding some faces I recognised and settling in to the Electra Room where the main track was due to start, with Inviqa's very own Rowan Merewood taking to the stage to deliver the opening keynote.

Rowan Merewood

Building Better Developers

After starting his presentation by pronouncing himself a tool, Rowan went on to highlight a number of key aspects to being a good and productive developer. Luckily for him, he was speaking to an audience who appreciate their tools and the content was well received. The huge cinema screen was a little odd at first, but once you'd gotten acclimatized it was a good addition to proceedings – I only hope it hasn't ruined my enjoyment of future conferences where such impressive AV equipment is not available. Every time Rowan added a new body part to his dream colleague it came with a twitter friendly summary, many of which did make it to twitter soon after - “Being passionate and authoritative is not the same as being an angry asshole” was one of my favourites.

Thanks to a prompt start there was plenty of time to decide what to go and see next, I decided I was going to head to track two. As I arrived it seemed many others had the same idea and I found myself taking the last available seat.

Alex Bilbie

API Driven Development: Eating Your Own Dog Food

Despite some minor technical hitches with the microphone, Alex Bilbie spoke confidently about some of the work he has been doing at Lincoln University. There were some good insights on why we should both develop well designed APIs, and then actually use them ourselves - if it's good enough for your clients, it's good enough for you, right? The content was kept quite high-level throughout and I actually took more from some of the periphery topics mentioned than the main topic, but I did enter the room already sold on the concept. I came out with a note to investigate Swagger for documenting my next API and Alex was also the first of the day to talk about measuring metrics and building dashboards, a topic that would recur throughout the event.

Bastian Hofmann

Measuring and Logging Everything in Real Time

A quick dash up the stairs back to track one to watch Bastian Hoffman talk about logging. I'd done some reading around Graylog2 based on an aside during a talk at the PHPNW conference but was yet to use it in anger, so I was pleased to hear more on the subject. Many more tools for me to investigate were introduced, including StatsD and Logstash which can, if used all together, create a robust centralised logging solution. Of course we mustn't forget that after collecting all this data we need a good way to see it. Creating yourself a dashboard using Graphite gives a great way to do just that. Bastian's suggestion to add a unique id to error messages to make it possible to identify the specific instance of what went wrong, when and which requests were related to it sounds so obvious now that I wonder why I haven't been doing it for years.

Lunch

Time for lunch and an unexpected surprise as everyone was handed a black plastic PHPNE branded lunchbox containing a sandwich, crisps and a variety of sweet treats. I've not seen this done before at a conference, and it seemed to go down pretty well. The lack of chairs added to the unusual lunch choice, as many of us sat on the floor and ate our sandwiches in scenes more akin to a summer picnic than a PHP conference. It was also during lunch that I discovered a personalised mug had been provided for me, and every other attendee, by one of the sponsors – another unexpected little bonus.

Fabrice Bernhard

Modernisation of legacy PHP applications using Symfony2

First up after the break was Fabrice Bernhard talking about upgrading legacy apps. Many of the points here were useful, but it did feel at times like a marketing pitch. It also seemed unnecessary to have included Symfony2 in the title, as most of the talk was around techniques and considerations for upgrading rather than the technical detail of implementation. Nonetheless the concepts were sound and the key message to upgrade legacy code module by module rather than a complete rewrite from scratch makes a lot of sense. The asides were interesting too, another mention for measuring business metrics and dashboards along with promoting the idea of regular deployments, regular here meaning many times a day, not the fortnightly or even monthly release cycles often seen in agile projects.

My decision to stay in track one for the next talk meant I had now committed to my chair for another two hours and I was certainly pleased with the cinema seating at this point, a lesser seat may have had me more concerned about the time ahead.

Lukas Kahwe Smith

Introduction to Symfony CMF

Despite being an official sub-project of Symfony I have to admit I had no knowledge of the Symfony CMF before this session. I was pleased to be introduced to it as it looks like an excellent option for any developer who needs to customise a content management system. Lukas was very accomodating to the crowd and even offered a choice of showing slides, a demo or the code. With only one hand raised for slides and a split between the other two options, we were treated to a quick overview of the system and a little bit of code. I came away eager to get a chance to play with some of the features and hope that going forward the project will catch on, as it looks a far nicer system for developers to work with than some of the CMS systems I've come across.

Thijs Feryn

Keeping The Cloud In Check

To close the day, Thijs gave a well judged closing keynote about “the cloud”, or as he put it “the internet”. Lots of amusing anecdotes helped sell his message, his insistence that nobody tweet the stories suggests that they may well be true too. The core message throughout was that there is no silver bullet solution to high availability and scaling problems, and the fact that many people believe “the cloud” will solve all of their problems is in itself a real problem.

Wrapping up

The post-talk conference giveaways were interestingly executed, as the winner of each prize was selected by a php script, not in itself a complicated idea, but it added to the event nicely. Finally, after a quick word from the sponsors and some difficulties operating a mac, which entertained the delegates enough that part of me wonders if they were scripted, everyone was given a free t-shirt as they left the Tyneside cinema to add to the goodies that had already been showered upon us throughout the day.

After heading out for some food I ended up back at the post-conference social, where a sponsored bar was in full force. Given that the money behind the bar did not run out until gone 1AM, I feel little should be said about the events of the social, other than I had some interesting conversations and I am appreciative to the organisers and sponsors for a very enjoyable day and night. I took something from every talk and I don't really have a bad word to say about any of it. I hope this does not end up as the only PHPNE, it is only a shame that next year a new venue will likely need to be found as I expect demand for tickets will be high.

                          

Building these apps has been a learning process for us so I thought it might be useful to share some of things I’ve learned along the way.

Test on Real Phones

In the world of iOS, things aren’t too bad, using Xcode and the iPhone emulator is fairly realistic. Even then however, there may be some nasty surprises if testing on real devices is left to the end of the project. For example, you might find that your animation looks great on the emulator but is painfully slow on an iPhone 3GS. Some features can't be emulated; for example if your app uses the accelerometer then there’s no way to emulate it (you can try shaking your laptop but it won’t get you very far).

When it comes to Android there’s a huge set of different challenges. The emulator is often slow, unresponsive and frequently flaky when using it to run the app with Eclipse. Consequently I used it very rarely for testing. Luckily deploying an app to real devices during development using Eclipse and the Android SDK is straightforward, and unlike iOS there is no requirement to have developer accounts to test your apps.

How does a project decide which devices and versions of Android to test on? There are a plethora of devices out there and the potential combinations of devices and Android versions is huge. It would be impossible to test on all of them, but as a minimum it is best to aim for at least one of each of the main screen sizes, so something fairly old and small like a HTC Wildfire, a medium size one such as a Samsung Galaxy S2, and a large one like a Nexus 4. This only scratches at the surface of the range of devices available, so the best thing to do is agree with the client on which devices and Android versions will be used for testing, and always make sure this includes the phone they use.

CSS3 and HTML5

One of the nice things about working on mobile apps is that you can use CSS3 and HTML5 without having to worry about providing fallbacks There will be no need for HTML5 shivs, no images pretending to be gradients, or workarounds for Internet Explorer. Sadly that doesn't mean everything will work exactly as you expect, here are a few things that might trip you up:

1. Some CSS3 properties can affect performance, box-shadow is especially bad for this, so if your animations are looking choppy try removing any box-shadows, it may help.
2. 3D CSS3 animations are hardware accelerated, but you can trick it into using hardware acceleration on 2D animations by adding:

   -webkit-transform: translate3d(0,0,0); transform: translate3d(0,0,0);

3. Bear in mind that while many CSS3 properties no longer require prefixes, these may still be needed for earlier versions of Android and iOS. For example box-shadow requires the -webkit prefix in Android up to version 4 and iOS up to version 5. The ever-helpful http://caniuse.com includes mobile browsers in its compatibility tables. It's also worth noting that older versions may use different syntax for background gradients, but generators such as http://www.colorzilla.com can provide these for you.

jQuery Mobile

jQuery mobile can be very useful and time-saving when developing apps as it’s easy to use and well documented. From a front end point of view the ability to just be able to specify the desired transition directly on a link is great, it looks something like:

<a href="index.html" data-transition="slidedown">

However, where we found problems was when you need to have more control over the look and feel of your app. On another app we ended up removing jQuery mobile, writing our own page loader and spending time creating and tweaking a flip animation to get it to fit the client’s requirements exactly.

Handy Tips, or "Things that took far too long to find by Googling"

Renaming your App:

1. Android: Edit the app-name in [project-folder]/android/[project]/res/values/strings.xml
2. iOS: In xCode, select your project under Targets, then in the Info tab update the Bundle display name

Force the phone to show the app in landscape mode:

1. Android: In [project-folder]/android/[project]/AndroidManifest.xml add android:screenOrientation="landscape" to the activity tag
2. iOS: In xCode, select your project under Targets, In the Summary tab make sure that either Landscape Right or Landscape Left is selected and the Portrait ones are not selected.

Hide the phone status bar while the app is running:

1. Android: In [project-folder]/android/[project]/res/xml/config.xml add
2. iOS: In xCode, select your project under Targets, In the Summary tab select Visibility - Hide during application launch in the Status bar section.

Opening external links in the in-app browser:

Without this a user can click on an external link in your app, and then can’t get back to the app, luckily the current version of PhoneGap comes with InAppBrowser which you can use with the window.open call. For example, you can open a URL using a command like:

onClick="window.open('http://google.com', '_blank', 'location=yes');"

Developing with PhoneGap

This has only been a brief overview of the learning process we have been through while developing these apps, but hopefully there will be tips here that are useful to other developers taking their first steps into app development. Using PhoneGap does comes with some limitations, but for our apps it has done the job and for web developers, especially front-end types like me, it’s a good way to get started and begin to learn about app development.

The BNY Mellon Boat Race app is available for Android and for iPhone.

                          

Setting up a good web application monitoring system can be tiresome, but it's well worth it. Without the monitoring tools the only thing we could tell is if our site is performing as expected or not. In order to improve the performance we have to be able to identify the worse performing user actions and profile them independently to pinpoint the cause. New Relic achieves that and more in just a few screens, all without manually adding any profiling code to your application.

New Relic is a real-time application monitoring service, providing various metrics about the performance of your production site, covering everything from application database queries through to the time it takes for the end-user to view a page. This data is then collected, post-processed and converted to simple and clean charts presented in the New Relic web interface. Since the New Relic agent has to collect and report the data, it does add some overhead to the application stack. Unless you're running a service that has to respond in a few milliseconds, however, the overhead added is minimal and is far outweighed by the value of the reports enabling you to detect and solve problems early.

This article covers both the basic functionality of New Relic (that can be used for free) as well as describing what the Enterprise version has to offer.

Getting Started

The New Relic installation is split into several distinct components:

• the agent component – a PHP extension, which collects the data and reports to a locally running New Relic daemon.
• the daemon component, acting as a proxy between the PHP agents and New Relic datacenters. The main responsibility of the daemon is to reduce the time of reporting the data to New Relic.
• New Relic reporting suite – the main New Relic website, where the data is presented for the user.

Both the agent and the daemon components are installed using the provided newrelic-install script. The script will detect the available PHP installations and deploy the agent extension to all of them.

Please refer to the official documentation for more detailed information regarding the installation and configuration of the New Relic package on your specific platform.

Basic Charts

“Forever. Seriously. That’s right. Free. Just the basics.” (newrelic.com)

While the free version of New Relic does not have all the bells and whistles of the Enterprise version, it does provide some basic, yet useful, feedback regarding your site performance.

Application Performance

If you are interested in an overview of how your application performs at PHP level, this is the chart to look at. It displays the average execution time of your PHP scripts in real time, split into separate layers by their execution type:

1. time spent to execute database queries
2. time spent in PHP
3. external web service calls

Depending on which of the application layers take the most time, different optimisation (or scaling) techniques can be applied.

1. The time spent in the database can usually be reduced by doing one or more of the following:
   • analysing the queries that are executed, ensuring that they are using indexes correctly or creating new indexes for them
   • caching the most frequently-accessed and computationally-expensive result sets from the database
   • optimising the structure of your application’s data using techniques such as database partitioning
   • scaling up the database layer (e.g. adding more slave nodes)

2. The PHP time is the time that your application is processing the data and good tactics for bringing this down could include:
   • caching intermediate results
   • optimising the code, using faster algorithms
   • adding more webnodes (if the hardware limits are reached)

3. A few things that can help to optimise the network operations:
   • cache everything that can be cached;
   • introduce an asynchronous job queue if possible.

Browser Performance

Even if the performance of your application is good, there is no guarantee that the users of the site will get it loaded within a reasonable timescale. The New Relic chart for browser page load time provides an overview of how your website is performing.

Similar to the application performance overview shown previously, this chart is composed of several separate layers:

1. Web application time tells how long it takes for your application to process the requests. More information about this layer can be seen in the application performance chart shown earlier in this article.

2. Network time is the time that is spent purely for the user request to travel to your application server and then for the response to reach the browser, disregarding the time spent in the application itself.
   To optimise this component one would need to review the network performance of your system architecture by considering the following questions:
   • do you compress the response sent to the site users (depending on Accept-Encoding)?
   • is it just that the response is huge and some of it can be loaded later?
   • is the bandwidth used reaching its limits and you simply need a faster network link?
   • would distributing your servers worldwide (see more on this later) help?

3. DOM Processing includes the time from browsers receiving the HTML until the DOMContentLoaded is fired. The work done by browsers is essentially a preparation to execute your JavaScript logic, thus, it prepares the DOM structure, preloads CSS (if there are any script tags after them) and JavaScript files.
   To optimise the site performance at this New Relic layer, there are several actions that can be performed:
   • make fewer requests to retrieve JavaScript and CSS files
   • minify and compress JavaScript and CSS files
   • set caching headers so that static files could be cached by the client
   • reduce the number of DOM elements
   • use a CDN for the static files

4. Page Rendering time is the time a browser takes to download everything needed after the DOMContentLoaded event; this is usually the time when all the images are downloaded. Similarly as for the JavaScript and CSS files, the list of actions available to optimise the delivery of static content includes:
   • optimising image files
   • using CSS sprites instead of multiple images
   • using correct caching headers
   • using a CDN to bring your content closer to the user (there's also a chart for that, shown later in this post)

An extended list of rules and how they affect your website’s frontend performance can be found at Best Practices for Speeding Up Your Web Site. Also, there are several tools that can help you to analyse the frontend performance such as Yahoo’s YSlow or Google’s PageSpeed.

More information about the Real User Monitoring functionality can be found at How Does Real User Monitoring Work? (New Relic documentation) or How we provide real user monitoring: A quick technical review (New Relic blog).

Throughput Charts

Even if your application is performing very well, it is only performing this way given the request rate at that time. As the number of users on the site increases, new bottlenecks “appear” – which can slow the overall usage of the site or even bring it down. As a result, the rate your application is handling the requests is at least as important as the time it takes for your application to send the response.

There are two separate throughput lines available – one for the browser requests and one the application. The browser throughput tells how many pages were requested per minute. Some of those requests may be served from cache before even reaching the application server, other pages may include additional application requests via Ajax. Thus – the two lines may be completely different and suggest different optimisation targets.

Apdex Rate

In addition to measuring your website performance in time, New Relic provides an Apdex score, which tells how many of your site visitors were satisfied, tolerating or frustrated by the response time of the application.

Once the target times for the browser and application servers are set, it will be used to calculate the Apdex ratio:

• satisfied requests are all requests that have completed in less than the target time (T), and “pull” the Apdex score towards 1.0
• tolerating requests are those which have taken more than T, but less than 4*T. These requests are given the score of 0.5
• frustrated requests are all the rest, and their Apdex score is 0.0

The main difference between using the Apdex ratio and the application response time, is that no one request (outlier) can affect the global ratio more than any other. This makes it a more scientific metric for the global overview of your site’s performance if your goal is to answer the question “what proportion of the site visitors see a page loaded quickly enough?”.

For more information about the Apdex score see the New Relic documentation about this metric.

Worldwide Site Delivery

Is your site performing well for local users? What about the users overseas? The Internet is really fast these days, however it is not instant. The further your user physically is from your servers, the longer distance the information packets will have to travel.

To get a glimpse of how your site is performing for different countries, you could look at the worldwide Apdex chart. New Relic also provides more detailed information for the enterprise customers.

Since the performance problem in this case is usually due to the global network speed, there is no fix that can be applied locally – you’ll need to bring your service closer to the user. Depending on your application needs one or more of the following measures can be employed:

• use a CDN to serve static files from local servers
• use local dynamic content caching servers for slower areas
• implement your service locally for the slower areas

Enterprise Functionality

“The Total Package!”

The basic functionality that New Relic offers for free can give us a lot of valuable insights about the global site performance. We can see what areas need more attention than others and this alone can save some precious time while optimising the site. Yet, it does not provide some of the (sometimes crucial) information; where exactly is the bottleneck?

In addition to the free Lite account New Relic offers two more plans (Standard and Pro) which extend the basic reports and introduce some new ones, allowing you to drill down to the root of performance problems quickly and efficiently.

Application Profile Traces

One of the best features offered to help debug performance problems is the comparison between different web transactions and the ability to see timed application traces of slow calls. New Relic provides charts similar to those described above for each web transaction type (provided that New Relic supports the framework you’re using). Also, a list of slow transaction traces is included with the detailed information.

There are already several tools available to profile your PHP code, such as Xdebug and XHProf. Xdebug is a really powerful development tool as well as offering profiling capabilities. XHProf is simple to configure and relatively easy to use, and there are also companion tools such as XHGui which make life even easier. So what is different about New Relic?

The code profiling trace that New Relic provides is a call tree with only Incl. Wall (absolute and relative) information. This tree alone is not very well suited for a generic code analysis since it does not provide the count of how many times a method was invoked, nor its total time during the application run. The power of it is that it is integrated with all other New Relic features and is easily accessible for a quick review once a slower transaction is detected. In addition to PHP code profiling, New Relic also provides a separate report for slow SQL statements, with their execution times and call counts.

With the help of these integrated traces, finding slower pieces of the application code is a straightforward task, helping to keep the focus on the site as a whole while still being able to detect problems and pinpoint them to the method level.

Compare with Historical Data

In addition to displaying the current state, New Relic also provides a comparison mode. When this mode is turned on, all the basic charts are affected – in addition to the current data they now also provide information from one day and one week ago. This mode is especially useful to show whether the site is performing any better (or worse) than before.

Scalability

How well is your website performing under load? The easiest way to answer that is to look at the scalability chart that New Relic provides. The chart plots the application response time versus the throughput.

This chart can quickly give you an idea about how well your website is responding given that there are a certain number of requests per minute. If the response time is constant as the throughput increases then your site is performing well. However, if you notice that the response time is increasing together with the throughput then it is time to take action. Finding the bottleneck using New Relic should now be an easy task using the database and application code profiling tools described above.

More information about this chart can be found in the New Relic blog.

Final Thoughts

New Relic is an amazing service to monitor your web application. It is simple and powerful – all the numbers are presented in such a way that a quick glance to the chart enables one to tell a lot about the site’s performance. In this blog post we have reviewed the common problems that New Relic can help us to detect and provided several suggestions of how to fix them.

Also, it is probably worthwhile mentioning that while New Relic is very good at what it does, it is a service to monitor your application and it usually works best if combined with a separate system to monitor the server resources or the performance of each service you’re using – understanding how the whole application ecosystem behaves is essential in order to build a stable and well performing web service.

                          

Getting Set Up

You can download Charles Proxy from the project website: http://www.charlesproxy.com/. Under the "Documentation" section you will find instructions for installing the tool onto whichever platform you use (Charles supports Windows, Mac and Linux).

Firefox Extension

Charles offers a Firefox extension from its download page that makes it very easy to start using your web applications with Charles immediately. Once you have the extension installed, simply run Charles and then enable the extension from the Tools menu in Firefox. When a web request is made in Firefox, the HTTP traffic involved is routed through Charles. This is a great way to get started and is the approach used to generate the first few examples in this article.

Inspecting Traffic in Charles

Charles isn't the only option for inspecting traffic, but it is at least as fully featured as using alternatives such as browser plugins such as REST Console or LiveHTTPHeaders, or capturing traffic using a tcpdump and/or Wireshark. Charles understands more types of traffic than just what comes through a browser, and has better support for storing, sharing, altering and repeating requests than Wireshark.

By default in Charles, when you start to capture traffic you'll be looking at the "Structure" tab, which shows all the requests that have been made, organised by URL. Each folder has a domain, then there are subfolders for each element of the path. The view ends up looking something like:

A more linear way to look at the traffic is to use the "Sequence" tab, which will show something like this:

Where to look next depends entirely what you're looking for. The filter box (below the sequence of requests and above the detail section) is very smart, including the ability to configure it to use a regular expression in its search, and will search all the fields shown in the grid. This allows you to filter down the results to just requests with a given response or using a particular verb, or to a particular domain or subdomain.

Once you have identified the request you want to see, check out the tabs in the bottom half of the screen. These typically show the following options:

• Overview: Some summary information about the request and response formats, sizes, time taken, and so on. This can be useful when comparing similar requests.
• Request: Full, detailed information about the request, headers, data sent. Look at the different aspects using the tabs along the bottom of the window.
• Response: Much like the request tab, the response tab has some very nice feaures, in particular its ability to nicely represent data formats such as JSON which can be viewed raw, pretty-printed, or parsed into a structure you can drill down into.
• Summary: Shows all the requests that were made as a result of this request, such as loading other assets to display the page, plus information about the overall size and time of these. Mostly useful when requesting a root page which loads in other elements such as images.
• Chart: Also mostly useful when requesting a page that loads other elements, this gives a timing diagram of things arriving as shown in the figure below:



Armed with this information, you can take a good look at what's taking the time in your application, which pages are pulling in which other pages, and so on. Once you have identified where a problem exists, we can use the following techniques to further diagnose or observe it.

Replay a Request

Charles offers the ability to repeat a specific request as many times as you want to. Either right-click on the request to see the options and choose "repeat", or press the blue circular arrows button on the toolbar to repeat the request. As a debugging tool, this is much nicer than having to fill in the same web form lots of times, or otherwise follow a tricky, multi-step process to replicate a bug.

The right-click menu also offers the ability to "repeat advanced", which allows the same request to be made to the server multiple times, including the option to specify the concurrency. If you are seeing weirdness under load or you suspect a race condition, then this feature is your friend!

Sharing Requests

It's possible to import and export sessions with Charles, in HAR (HTTP ARchive) format, which is supported by other tools as well. This is superbly handy when one developer is seeing a problem and wants to show another one what's happening without the aforementioned tricky process to reproduce the bug. Simply export your session from Charles and add it as an attachment in the bug tracker; whether it is you or someone else that picks this up, it makes it very easy to replicate the problem and be able to begin working on a fix.

Throttling Requests

This is a really good use of a local proxy; to throttle the traffic going through your browser in order to simulate a particular setup or network speed. For websites where it is likely that the user is either using a mobile phone or is connected via a mobile data connection, this gives us, as developers, insight into how the user experience will be in that scenario.

Charles offers a few options for speed restrictions to simulate different kinds of network connection, shown in the screenshot below:



It is also possible to limit throttling to just a particular set of hosts, so if for example you know that one particular 3rd party may provide slow connection speeds in some situations, then you can narrow that down using these settings.

Throttling is a very important step in testing mobile applications, especially when working with either mobile sites or with app simulators, since it gives a more realistic sense of the user experience. I lead an open source project (http://joind.in) which is used mostly at conferences, and conference wifi is usually fairly awful – so I throttle down to old modem speeds to give myself a sense of experience when developing that site.

Another feature of Charles which is relevant to this area of development is that it can disable caching. This enables you to reproduce the experience that a new visitor to your site will see. To achieve this, Charles changes the headers in the HTTP traffic, a feature that we can take direct advantage of ourselves.

Changing Requests and Responses

Sometimes, the request and responses that are made by an application aren't exactly what we need them to be. This can be useful for playing tricks such as redirecting traffic from a mobile app which assumes a particular URL for its API backend, or injecting a specific header to be able to trace your request in the server logs or to enable debugging for requests coming in via this route.

The rewrite functionality can be applied to all requests, or just a subset, and multiple rewrites can be configured at one time. They are encapsulated into rule sets, so you can save the settings of these, and then enable/disable them at will. For the rewrites themselves, there are yet more choices; here is the input dialog for creating a rule, to give you an idea of what's possible:



As you can see, the options here are endless. There are any number of reasons why you might need to add things, but these are my most common use cases:

• Injecting extra headers, either so that I can track a journey through the site because the additional information gets added to a log file, or to enable additional debugging.
• Injecting response headers to see how a client would react on receiving these from a server. HTTP supports the use of custom headers, so in theory you can use any header you like so long as your server knows how to understand it, and the RFC (see RFC 2616) recommends picking a prefix. In practice, it's common to use X- as the prefix for these "invented" headers.
• Modifying headers to alter a User-Agent header to chase down a particular bug.
• Setting Host headers – easily the most-used, mostly when using other machines or devices on a network to hit a local development virtual host with a different domain name than expected by the application (more on this in the next section showing how I develop with mobile devices against my local development machine via Charles).
• Adding a rule to do a find-and-replace on the body of a request or response is also very useful at times, for example to rewrite links.

These are very useful tools and the biggest advantage is that you don't need to recompile or reconfigure either the client or server to make these changes; Charles can intercept the traffic and make the changes on the fly. When developing a mobile app, where you might want to change whether it hits live or development copies of the back end API for example, this is a very quick and easy way to do it. It also protects against hardcoding these changes into your application and then finding that they get erroneously committed and deployed.

Mobile Debugging and Re-routing

Charles can act as a proxy for other devices to route via the machine it runs on, allowing us to observe and operate on traffic to clients other than the Firefox plugin. To get set up is fairly straightforward using the following steps.

First, check the IP address of your machine, and check or set which port Charles will listen on by looking under Proxy, then Proxy Settings. By default this is 8888, and in this example my machine has the IP 10.1.0.121.

Add either the machine hostname or IP to the device to proxy under the wifi settings for the shared network. I'm using an Android phone as I've been looking at the updated Android app for joind.in, although do be aware that some versions of Android (stock ROMs in the 2.x series, also known as "bug 1273") do not allow you to specify a proxy when adding network settings.



When you try to make a web request from the phone now, Charles will alert you that something is trying to proxy through, and give you the opportunity to allow or deny the access:



While working with the mobile app, this is a very easy way to observe the traffic being sent and received, and to spot any errors that might be silently ignored or not handled by the application (particularly as my application is in very early stages).

At this point we have the opportunity to perform all of the filtering and altering of traffic that were seen earlier when making web requests. In this scenario, sometimes I want to try the mobile app against my local development version of the API. To do this, I have a rewrite that simply routes anything for http://api.joind.in to http://api.joindin.local. This is in the hosts file on my machine, so it knows that domain is on the local host. Here is the rewrite rule in Charles:



When I tick and untick the "joindin-api-dev" box, I interrupt traffic to the live API, and instead send it to my local version of the API, which contains sample data and possibly local modifications. This is such a powerful feature when doing mobile development, because the data sources can be swapped without any need to restart the app, and certainly without changing or rebuilding it. This tactic is useful both when the app can't be changed, and also when it's simpler to intercept its traffic for testing.

Charles and SSL

In the interests of saving the best feature until last: Charles will also work with SSL traffic. SSL traffic is usually very difficult to debug, by design, because nothing other than the client and the server taking part in the transaction should be be able to see the traffic. An encrypted channel is established and everything is passed over it. So how can Charles intercept it?

What actually happens is that Charles executes a classic "man in the middle" attack. The encrypted channel gets set up between Charles and the remote server using the "real" SSL certificate. Charles then uses its own certificate to sign a second encrypted channel between Charles and the browser or client. In order to protect you against man-in-the-middle attacks, your browser won't accept a certificate that isn't from a trusted Certificate Authority unless you specifically grant permission for it. This is fairly common on development servers where it doesn't make sense to pay for an official SSL certificate, so you may have seen these types of untrusted certificate warnings in such scenarios.

To use this feature, first of all you will need to enable Charles to proxy SSL traffic, by ticking the box in the Proxy Settings screen. You should also specify which domains should have their SSL traffic proxied.



As we already mentioned, Charles then intercepts the traffic, using the remote site's correct SSL certificate for the connection between there and Charles, then an untrusted one between Charles and the client. This means that when you enable this in Charles, your browser will tell you that there isn't a valid SSL certificate for the remote site.



Confirm the security exception and you can now proceed as normal with all the features mentioned in the earlier sections of this post. Charles isn't the only tool to offer this functionality, but it's one of the easiest to get up and running with. When testing or debugging for SSL setups, this is a very valuable addition to the toolbox, and one that I hope you'll find useful moving forward.

Charles, Web Debugging Proxy

This article aimed to give you an overview of some of the most useful aspects of Charles for both web and mobile developers, as well as some pointers on how to actually apply the techniques covered here. All the requests, settings and configuration in Charles can be exported and shared, which is a huge help when using this tool across a team.

Other good products are available and you may also choose to make use of these. We also use Curl for being able to make many different kinds of web requests and to easily share them by pasting the commands into the bug trackers. Another alternative if you only need to inspect (non-SSL) traffic is Wireshark, as we mentioned earlier in the article. Both Curl and Wireshark are free, multiplatform tools.

Charles' tag line "web debugging proxy" is pretty accurate, making it indispensable in many situations. If you've got a Charles tip to share, or want to tell us how you make use of the techniques shown here, please do leave us a comment, we'd love to hear from you!

                          
]]> http://techportal.inviqa.com/2013/03/05/manipulating-http-with-charles-proxy/feed/ 2 http://techportal.inviqa.com/2013/02/26/phpuk-2013-conference-day-2/?utm_source=rss&utm_medium=rss&utm_campaign=phpuk-2013-conference-day-2 http://techportal.inviqa.com/2013/02/26/phpuk-2013-conference-day-2/#comments Tue, 26 Feb 2013 18:07:40 +0000 Tom Kitchin http://techportal.inviqa.com/?p=4411 It’s the start of a new day, and off we go to the Brewery for the second day of PHPUK 2013. Day 1 has been covered with aplomb by Javier so I’ll not go into details here, but suffice it to say that it was my first day of my first conference and I had an excellent time wandering without the hint of a clue, eyes filled with the misty wonder of a newborn. This, however, is about day 2, with a day’s worth of experience under my belt, the confidence of a conference veteran, and the bleary eyes of an early morning after a late night’s social.

The day started off well, with an excellent breakfast which I’d mostly missed out on the day before. A colleague had extolled the virtues of the porridge - the porridge - some four hours after breakfast had passed on day 1, and any breakfast able to make porridge so memorably exciting had to be tried. It was, I think, possibly the best porridge I have ever tasted. A strange thing to score a conference on, but there we have it. Moving on...

Talks - Saturday morning

We started with a keynote delivered by Martijn Verburg, The Diabolical Developer. He seemed human enough to me, but I suppose he might have been in disguise - you never can tell with Java developers. In any case, a confidently delivered talk on the subject of best practices disguised as exhortations to protect job security. Team meetings? Ha! Avoid your co-workers and manager at all costs, lest they try to find out what you’re doing or - horror of horrors - try to help. The thrust of the talk, as I understood it, could be summarised as "best practices are there to serve the developers, not the other way around". Careful analysis of team practices to make sure none of them are hindering more than they help is clearly worthwhile, and the core message of "the developers and the project are the most important thing" is worth restating regularly. Unfortunately, regular but unmarked switches between irony and serious speech made the talk a little confusing, weakening the argument.

For my first talk decision I headed off to Alvaro Videla’s talk RabbitMQ Hands On, which was an introduction to - you guessed it - the messaging system RabbitMQ. My experience with messaging systems is limited to Gearman, which I found a little unwieldy for me, so I was pleased to receive an excellent introduction suitable for newcomers. Alvaro gave a strong talk aided by an excellent RabbitMQ simulator of his own creation. Whether you’re a RabbitMQ greenhorn or a veteran I strongly recommend looking out for his simulator - he plans on open sourcing it soon, but until then there’s a handy introductory video which is well worth watching. I found the talk both informative and encouraging, and has left me watching out for an opportunity to make use of RabbitMQ in a project some time soon.

My second talk was Scaling with HipHop, delivered by Sara Golemon. This was, I think, the most enthusiastic talk I attended over the entire conference. Quite infectious, really. I walked in with a passing knowledge of HipHop ("it’s sort of a PHP to C++ compiler for Facebook, right?") and walked out quite dizzy with knowledge. Sara came across as both knowledgeable on her topic and deeply passionate about it, a combination which is always a joy to find. HipHop itself is rather more than I had thought, effectively a highly tuned replacement PHP engine which analyses and converts PHP to native bytecode for storage in a database, providing an astonishing 500% performance boost over PHP 5.2. Throw in the additional PHP features, the in-house-developed streamlined web server and even the in-house IDE and it becomes clear how Facebook can handle around one billion active users.

One billion active users. One billion. Active.

I know this is kind of old news, but just think about that for a few moments. For reference, current estimates put us at around 7 billion people on the planet.

Huh.

Lunch

Anyway. Lunchtime! Lunch was up to the standards set by breakfast. Not much more I can say about this, except for a brief comment on the strange image of a room full of developers holding plates with those funny cup holder clips on, fighting over about twenty people’s worth of table space and even fewer seats.

I took the opportunity to cheerfully obliterate the current record for the Inviqa/SensioLabs Bigtrak programming race... which would be more meaningful if I hadn’t spent some of the day before practicing. My honour demanded that I keep my time to myself, as did the judges who sadly recognised me perfectly well. So much for fame and fortune.

I didn’t actually attend the talks immediately after lunch, as I was feeling quite overwhelmed by the veritable torrent of knowledge I’d received that morning. Instead I sat down in the Hackspace and investigated (read: googled) some of the technologies I’d heard about, as well as having a chat with the chaps over at the LiveRebel booth. Worth checking out, incidentally - looks like quite a neat deployment management solution.

Talks - Saturday Afternoon

On, then, to the fourth talk: You Can’t Optimise What You Can’t Measure. Juozas "just call me Joe" Kaziukėnas handled this one with the message "logs are useless." Don’t even write them. I recognise the knee-jerk reaction we all have to that, but he made a good case - and I should clarify that we’re referring to production servers here. How often do we even read our logs? Instead he made a strong argument for good metrics-based data gathering, passed to suitable analysis tools. If we can be automatically warned whenever the metrics start to veer off their expected state, not only can we react to failures and bugs but even user experience issues we would otherwise have missed. Not so sure about "data doesn’t lie," though. Data might not lie, but we have to convert it to information to use it, and we can certainly bring a few of our own lies in (quite unintentionally) on the way. I found his argument persuasive, in any case.

For my final talk I attended Realtime Web Apps in the Wild with Phil Leggetter. This was primarily about event-driven web applications, as advancing technologies have finally allowed us to implement a "true" real-time web. The talk actually had a set of interactive apps woven into it, accessed by attendee browsers from smartphones to laptops - a fun addition to the usual style of a talk. A good discussion of available technologies (with a commendable lack of bias from the Pusher developer) and an informative if light coverage of the history of hacks, techniques and trickeries used to emulate realtime work in the past. It’s rather subjective, but I found the talk to have less impact than any of the others I attended, though. Perhaps this talk would have had a greater effect on someone who works more closely with near-real-time web apps - personally, I’m now aware of the technical differences but I struggle to really sense the difference between the old hacks and the new approaches favoured by the talk. Definitely worth a look if that’s your focus, though.

Last (but by no means least, as they say) we had the panel. Day 2’s panel had a much better feel than the first day’s, with some genuine enthusiasm and good humour displayed by a panel consisting of Aral Balkan, Ian Barber, Sara Golemon, Beth Tucker Long and Martijn Verburg. The topic was Open Source in Action, and to me the highlight was some discussion of previous open source projects by the panelists which had achieved things, or changed things, or ended up in strange and unexpected places. If you needed a reminder of why open source is worth our time and effort and our creative energies (and I discovered I did), it was worth hearing. It’s because you can write a piece of code for yourself, just because you need it, and throw it out on the web just because someone else might need the same - and in the end that code you’d have written anyway makes someone’s life easier, or ends up in places you never expected. A creative industry indeed.

Do I need to cover the post-conference social? I thought not.

Conclusion

So we come at last to the wrap-up. I had a great time at the first conference I’ve attended and hope others felt the same. I listened to and spoke to interesting, enthusiastic people, both speakers and attendees, and more than anything I feel like it brought home why I got into software development in the first place.

Because we can create amazing, interesting things that people need or like or enjoy or didn’t even realise they wanted or don’t need at all and share them with the world. Now, if you’ll excuse me, I have a few new projects I need to think about.

                          
]]> http://techportal.inviqa.com/2013/02/26/phpuk-2013-conference-day-2/feed/ 3 http://techportal.inviqa.com/2013/02/26/phpuk-2013-conference-report-day-1/?utm_source=rss&utm_medium=rss&utm_campaign=phpuk-2013-conference-report-day-1 http://techportal.inviqa.com/2013/02/26/phpuk-2013-conference-report-day-1/#comments Tue, 26 Feb 2013 15:32:20 +0000 Javier Lopez http://techportal.inviqa.com/?p=4347 Last week, the 8th PHPUK Conference took place. Due to a previous commitment I could only attend Friday's session so if you think I am missing something, do not hesitate to leave a comment. Here is my personal point of view of the PHPUK Conference this year.

Talks

The conference opened with Aral’s keynote “You are a designer”. I have to say that the talk was just brilliant, probably one of the best talks I’ve ever been to. Unfortunately, in my humble opinion, the opening keynote for one of the biggest events on the PHP scene should be related to PHP. I know this is a matter of taste and I actually talked to a lot of people who thought it was an excellent opening.

Event Stream Processing in PHP

The next talk in my agenda was “Event Stream Processing in PHP”, by Ian Barber. I couldn’t have been happier to have picked this talk. Ian started from zero, introduced the basic concepts of event stream in an excellent way and went on to show us some code. This is the kind of talk that puts an idea in the back of your head that months later will pop out when you realise that you need it in a real life project.

Looking ahead: PHP 5.5

David Soria Parra, in “Looking ahead: PHP 5.5” introduced some of the new ideas coming up with the latest version of PHP. He announced that PHP is going to change its release cycle and is going to be bundled with the following new features:
- generators
- the password API
- DateInmutable
- ::class
- finally, mysql extensions are deprecated (At this moment he pointed out that WordPress core developers are going to have a lot of fun due to the intensive use of this function in the code, ;))

Probably, the best piece of news for the PHP community is that Zend Optimizer is likely (but not confirmed yet) to be included into the core.

You can read more about the new features on php.net.

It was quite shocking for me that when he was leaving the stage, he said he was eager to answer any questions throughout the event, but please, “don’t ask me any PHP code question, I am really bad at it”. It got me thinking and I realised that, ironically, he´s more of a C/C++ developer.

API Design: It’s not rocket surgery

Dave Ingram was explaining everything you should bear in mind when building an API. The talk was quite accurate but I would have added more spice in order to make it a little bit more dynamic. Anyways, with no hesitation, if I have to build an API in the future I will try to get back to these slides, I am pretty sure it’s going to be quite helpful going through all this valuable information.

Pair Programming, TDD and Other Impractical Things

Marcello Duarte, Head of Training at Inviqa, exposed with quite a sense of humour some of the new TDD practices. I really enjoyed the talk, not because it was mind blowing or there were new concepts I'd never heard of before, but how he managed to dazzle everyone in the room by exposing all these ideas from a completely different point of view.

If you are a newcomer to all the TDD philosophy, I really encourage you to check the slides for this talk.

Refactoring to Symfony Components

I closed my first and only day at the conference with this talk by Michael Peacock. Throughout the talk Michael showed us how he managed to rebuild an old legacy project using some of the Symfony Components (namely Routing, ClassLoader, Yaml, HttpFoundation and EventDispatcher) and some other third party libraries (Twig, Pimple and Swiftmailer). In my opinion, his strategy to rebuild the project was quite clever and showed us how sometimes you can’t follow the path marked by some of the biggest frameworks but you can still take advantage of all this code.

You can check all the slides of the event on joind.in

Venue

The conference was celebrated in The Brewery, in the heart of the City of London.



I first discovered this site last September, during the SensioLabs UK Conference and I have to admit I really like it. It’s really well located and it has an English style that makes it attractive for foreigners like me.

The talks were split into four different rooms. The keynotes and the panels took place in the main auditorium. The acoustics of this massive room are impressive. Moreover, for the people sitting at the very end of the room there were several screens broadcasting the slides.


(Picture courtesy of @aral)

The rest of the talks were running in three smaller rooms but they were spacey enough to host 200 people each. There was a room that was considerably smaller than the others but it was amended with a complimentary room where the talk was broadcasted live, and I have to say that the quality of the sound and the video was really good.

The only “but” had to do with the main track where the bottom of the slides were quite close to the floor and it was hard for the people at the back of the room to see.

On a side note...

Even though the PHP UK Conference is a British conference, you can’t lose sight of the fact that there are a lot of non-native English speakers attending. I really liked the fact that none of the speakers had a broad accent, actually, the delivery of their speech was quite neat. I am not sure whether they were making a conscious effort but I’d like to thank them all for this.

Also, the food was quite good. I am used to attending the sort of geeky conference where you get served pizza or cold sandwiches. Having a proper hot meal was great and sets the conference at a professional level, as it rightly deserves.

Summing up

It was a great conference. Everything was great, from the talks and speakers, to the organisation. I have to own up this conference is becoming a must in my diary and I am pretty sure I’ll go next year.



                          
]]> http://techportal.inviqa.com/2013/02/26/phpuk-2013-conference-report-day-1/feed/ 1 http://techportal.inviqa.com/2013/02/19/getting-started-with-git/?utm_source=rss&utm_medium=rss&utm_campaign=getting-started-with-git http://techportal.inviqa.com/2013/02/19/getting-started-with-git/#comments Tue, 19 Feb 2013 14:30:20 +0000 Kacper Gunia http://techportal.inviqa.com/?p=4329 On a daily basis probably each and every software developer spends a lot of time collaborating and communicating with other developers. There are several tools that simplify this communication (or at least should). One of those tools is a Version Control System. Not so long ago, a new tool appeared on the horizon, called Git. It quickly became a very popular tool in the open source community. Though the ideas behind Git are very basic, becoming a master is not easy! It takes some time, especially for someone who has had the "pleasure" of working with other systems like SVN or CVS previously. If you are starting with Git or switching from another tool, then this post is an aggregation of ideas, not a full-featured tutorial. If you are already an experienced user, perhaps you will consider sharing your knowledge in the comments.

Learn from the Best Sources

There are several places where you can read and learn about Git, but it is hard to know which resource is best. In my opinion the best source of Git knowledge is a free book called "Pro Git" (http://git-scm.com/book). The second is an online tutorial called Git Immersion. If you feel you missed something or want detail on something specific, then there is always the official documentation.

Pick a Workflow and Strategy

Git is a great tool that allows you to work in many different ways. The possibilities and flexibility that you can achieve with it are a great thing. If you want, you can even use it in exactly the same way that you were using SVN (but there is no real point in doing that). To simplify team communication and avoid conflicts, it is good to follow some rules and standards created by other developers. I would recommend you read about GitFlow and GitHub Flow. Then you will be able to pick a strategy that suits you (and your team).

Customise Git

Git is only one of the many tools that you are using, but you will be spending a lot of time with this tool, so it is worth taking the time to customise it to your own needs. Most of the useful options were described in the Pro Git book, and I have also included below some basics from my configuration. To use these, you will need to edit the config file placed in your home directory. For example:

$ vim ~/.gitconfig

You can then copy, paste and customise the following lines as you wish:

[color] diff = true [user] name = Name Surname email = your@email.com [alias] co = checkout fa = fetch --all ci = commit st = status br = branch last = log -1 HEAD unstage = reset HEAD amend = commit --amend staged = diff --cached unstaged = diff

Another great approach is to steal the best configurations provided by my colleague Alistair on his GitHub account https://github.com/alistairstead/dotfiles/blob/master/.gitconfig. Alistair has taught me a lot about Git, so trust me when I say it is worth following his way of using it.

Extra tip: configure your Bash prompt to display current branch and other related information, using the instructions in http://asemanfar.com/Current-Git-Branch-in-Bash-Prompt.

Introduce Gamification

If you like to be rewarded for your work you might be interested in a project called Git Achievements. This project allows you to receive points for using Git on a daily basis, and you can challenge your friends to do the same! To get you started, my result http://cakper.github.com/git-achievements is waiting to be beaten by you ;)

Just Do It

The best way to learn is to begin - right now! Create a testing repository, play with it and don't be scared to break something. The skills you learn by practising now will be very useful in the future. Not everything will be easy from the beginning, but to help you along your way here are some tips for solving common problems:

To edit the most recent commit message:

$ git commit --amend -m "new commit message"

To revert an uncommitted file to the previous version:

$ git checkout HEAD path/to/file

To delete your last commit:

$ git reset --hard HEAD~1

To create a new branch from the current one (and move all uncommitted changes):

$ git checkout -b branch-name

To check out a remote branch from the origin repo:

$ git checkout -b branch-name origin/branch-name

If you want to learn some more serious stuff, a great way to do that is to join an open source project on GitHub. In my opinion it is the best place to acquire skills from other people, and also a good opportunity to use your own skills to contribute to a project used by many people.

I hope this short post will be a shortcut and allow you to kick off your experience with Git in a painless way. Having good Git skills will improve the way you work, and in time I think you will come to share my impression: this is such an easy and beautiful tool.

Leave your tips and share your experiences in the comments.

                          
]]> http://techportal.inviqa.com/2013/02/19/getting-started-with-git/feed/ 4 http://techportal.inviqa.com/2013/01/08/2012-a-year-in-php/?utm_source=rss&utm_medium=rss&utm_campaign=2012-a-year-in-php http://techportal.inviqa.com/2013/01/08/2012-a-year-in-php/#comments Tue, 08 Jan 2013 08:38:32 +0000 Ben Longden http://techportal.inviqa.com/?p=4307 Another year has passed, and yet again we find ourselves reflecting on what has happened in the world of PHP over the past year. Let's take a moment to look back over our 2012 predictions and see how close we were!

Framework Convergence

In 2011 we said that major frameworks would start to converge on similar strategies for basic functionality. The PSR-0 standard defined a common autoloading strategy, and the release of Composer in March 2012 has really boosted the adoption of this standard across the most popular frameworks. The release of Symfony 2.1.0 in June brought in Composer as the dependency management solution of choice and the first stable release of Zend Framework 2.0 followed in September, also using Composer for autoloading and dependency management.

In March, the Drupal project announced that version 8 of their platform (currently due for release mid-2013) would be completely rebuilt on top of the Symfony components, an exciting development and certainly one which has started to bring the Symfony and Drupal communities together at conferences and other community events.

It does appear to be the case that projects, even up to the framework level, are starting to move in the same direction, but the choice remains: Zend Framework 2 or Symfony 2? I think we can claim a point for this one!

Dependency Injection Containers

At the time we made this prediction, Symfony 2 was already in the wild and Zend Framework 2 was under heavy development with alpha versions available. We saw further releases of these two big hitters through 2012 and DIC (Dependency Injection Containers) have been a popular topic throughout the year at conferences. It's enabled practitioners to create testable, reusable code and has become a standard practice amongst practitioners of good PHP.

Serious Caching

We said that caching would be a big topic of 2012 as large enterprises and other organisations start to leverage open source solutions for their high traffic web applications. Almost all projects that we have started in 2012 have had a varnish layer designed in from the start to achieve the high performance that our clients expect, and Edge Side Includes (ESI) has started to see more use in new applications - putting caching proxies, of which Varnish is one, at the core of the platform.

We'll claim a point for this one as we have seen caching being a really hot topic in 2012, and one which we expect to continue to see in 2013. Varnish has seen an increase in usage during 2012 of 39% (see http://trends.builtwith.com/Web%20Server/growth#!oneYear)

Trending

Composer

One of the most popular topics that has come up at conference talks in 2012 is Composer. We've mentioned it already in this write up, but if there was a top 10 for what has changed the face of PHP over the year, this would be #1. Not only has Composer been a conference talk topic in its own right across many events this year, it has also featured in talks about many other technologies. Support for Composer has become something that people working on open source PHP project libraries are proud to announce as a feature of their product - and quite rightly so! It has already become something that's almost expected for all PHP libraries - and pull requests on GitHub to add support for it across the board have become commonplace.

Best Practice

The old argument between PHP users and users of other languages has raged on. We are proud of PHP's roots in server-side templating, but the language has evolved into something completely different today. Because of this history, the PHP manual is focussed on the quick way of getting something done in PHP rather than what is necessarily the best practice, and 2012 saw the launch of a really great resource for developers: http://www.phptherightway.com/. It is a collection of recognised "best practices, options and good information" on using PHP and is a highly recommended read for all users of the language. The recent addition of comment voting on the PHP manual will hopefully promote good information over poor practice in the official manual over time, which will only benefit us all.

PHP5.4, PHP5.5, PHP5.2 and PHP5.3 (in that order...)

The 5.4 stable release of PHP came out in March, and since then it has seen 10 maintenance releases - bringing us to the latest stable release of 5.4.10 (at the time of writing).

Towards the end of 2012 we also saw the first alpha releases of PHP 5.5 - the next major release of the PHP 5 language. This will bring new features such as generators (simple iterators with the addition of the yield keyword in functions), the "finally" keyword (try, catch, finally), a new and redesigned password hashing API and various other improvements to the language core. It's worth mentioning that the current release is still alpha and new/deprecated features are still subject to change.

The w3techs usage statistics reports that PHP 5.2 is still the most popular deployment of PHP out there in the wild with 52.1% of all PHP deployments running on that version. The language hit end-of-life status back in 2010 and users of sites built on this old technology have been encouraged to upgrade to 5.3 as a matter of urgency.

PHP 5.3 is in second place with 42.9% of deployments. The PHP group announced that 5.3 would reach an end of life status in March 2013 - with these users being encouraged to upgrade to 5.4. It's important that existing users of 5.2 start to look at upgrading as the work required to jump two or even three major versions will quickly become problematic. A small minority of sites are still stuck on PHP 4 and PHP 5.1, but we won't talk about those...!

PHP Closures

How did we code in PHP before the 5.3 release brought us closures? From being able to write PHP code directly into core functions like array_map() et al, to seeing frameworks like Silex gaining popularity, built on Symfony2 components and leveraging the coding style that having support for closures allow, this welcome language addition has made it easy for us to write better PHP.

For example how ugly would the code of the Pimple project be, but for the support of Closure within PHP?

Predictions For 2013

Now we come to the part where we try to predict what is going to happen in the land of PHP during 2013.

Framework Convergence ... Again

We predicted it last year and some massive steps were made towards achieving this goal. We think that 2013 will bring more efforts to be able to share modules between frameworks through the establishment of defined interfaces. These will be implemented by the framework core (a common approach to things like logging, authentication, etc). It only makes sense to build on what has already been achieved in 2012 with Composer and Packagist (the package repository that Composer uses).

Content as a Service

It's becoming more and more common for clients to ask us for a way of being able to create content and share it across many different sites and software platforms. Currently, the most popular CMS products force you to display the content you create through the templating system that is in-built to the platform you are using. We think that in 2013 we will start to see solutions for allowing a content repository to be able to push its content out to many different places - especially as more people consume data on a mobile platform.

Dependency Injection Containers Mark 2

With the previously-mentioned greater adoption of DIC within frameworks, more and more people have made use of DIC to resolve dependency management within their code. However this has lead to misuse by some and problems of an entirely different nature, especially within code libraries. As the wider community learns from these experiences good practices are formed. Dependency Injection needs to be considered when designing OO software rather than just ignored and left to the DIC.

BDD with PHPSpec and Behat

XUnit style testing frameworks seem to be losing favour within the community and the adoption of Spec frameworks and Behaviour Driven Development is set to increase in 2013. With the initial alpha release of PHPSpec 2 in October 2012 this alternative testing framework has given the PHP community a fresh perspective on testing and where it fits into the development process. This lean framework by design focuses on being a tool to help and support writing well structured OO code rather than provide mechanisms for validation. Used in conjunction with Behat, PHP now has the tools needed for full stack BDD. We predict a much greater adoption of these tools and BDD over 2013 and beyond.

Use and Misuse of Traits

Once PHP 5.3 reaches end of life status in March, framework and library developers will start to create releases with PHP 5.4 as a minimum dependency, allowing them to make use of traits. We anticipate that this will result in a great deal of code deletion from frameworks as multiple horizontal inheritance will allow a lot of boilerplate code from iterators (for example) to be removed and placed into traits.

This sounds like good news, and it is. However with great power comes great responsibility and this particular language feature is likely to be both used and abused! The community is always quick to learn from its mistakes however and we're sure that by this time next year traits will have had lots of good and some bad press.

PHP and You in 2013

What is on your horizon for the coming year? Is there anything we should have seen coming and didn't mention? Leave us a comment and let us know - and good luck with your PHP this year!

Contributors

Ben Longden
Alistair Stead

                          
]]> http://techportal.inviqa.com/2013/01/08/2012-a-year-in-php/feed/ 15 http://techportal.inviqa.com/2012/11/01/global-git-ignore/?utm_source=rss&utm_medium=rss&utm_campaign=global-git-ignore http://techportal.inviqa.com/2012/11/01/global-git-ignore/#comments Thu, 01 Nov 2012 09:48:19 +0000 Alistair Stead http://techportal.inviqa.com/?p=4288 I was recently reviewing a number of projects on GitHub and I was amused by some details about people that can be inferred by the content of a project's .gitignore file. I could identify the owner's base OS and their preferred IDE - even some of the specific development tools or system plugins they have installed.

If I worked for a marketing company, this could be a source of data to be mined to sell expensive alternate software. After all, between your GitHub identity and the .gitignore content, targeted contact lists could be derived. I hope that no marketing executive reads this otherwise I apologize in advance!

Projects should have .gitignore rules committed as part of the project. This is of course functionally correct. However each individual developer might want to consider setting up their own system-wide ignore rules, so that project-specific files remain project-specific. You can then add rules to ignore the configuration files from your preferred IDE that no one else in the project uses.

There are many very valid reasons why you should do this:

• The project leads don't have to do this for each project or change it when you as a contributor move to a new IDE.
• You can set-up your system to function they way you want without it impacting the project or others.
• Some potentially unsafe* commands such as 'git add -A' become safer as you will not commit sensitive or user specific files.
• If you're sending pull requests to Open Source projects, they will not be declined due files that should not be present.
• Reduce the need to duplicate rules between many projects for non-project files.

Okay this sounds good, how do I do that?

Firstly let's create a file on your system that will hold these system-wide ignore rules.

$ touch ~/.gitignore-global

Then add the appropriate rules that you want to be ignored across all projects. Below is the content of my own .gitignore-global file

# Compiled source # ################### *.com *.class *.dll *.exe *.o *.so   # Packages # ############ # it's better to unpack these files and commit the raw source # git has its own built in compression methods *.7z *.dmg *.gz *.iso *.rar *.tar *.zip   # Logs and databases # ###################### *.log *.sqlite   # OS generated files # ###################### .DS_Store ehthumbs.db Icon Thumbs.db .tmtags tags vendor.tags tmtagsHistory *.sublime-project *.sublime-workspace

Into this file you should add matches for things like system cache files or system and IDE configuration files.

Once you have this file created you will need to edit your .gitconfig file to add this additional configuration file. This can be done by editing the file manually or by running the command below:

$ git config --global core.excludesfile ~/.gitignore-global

That's it! You now have system-wide git ignore rules that will prevent you from needing to duplicate these rules into each project. This alone is a good enough reason to do this; the others are just a bonus.

* By unsafe I mean, if you don't appreciate what the command or flag does. I often see newcomers to Git use this shortcut with unexpected results.

                          
]]> http://techportal.inviqa.com/2012/11/01/global-git-ignore/feed/ 0 http://techportal.inviqa.com/2012/10/03/restfest-2012/?utm_source=rss&utm_medium=rss&utm_campaign=restfest-2012 http://techportal.inviqa.com/2012/10/03/restfest-2012/#comments Wed, 03 Oct 2012 07:27:08 +0000 Ben Longden http://techportal.inviqa.com/?p=4263 I was fortunate enough to be able to attend RESTFest in Greenville, South Carolina on the 13th, 14th and 15th September 2012. RESTFest is an un-conference organised by Mike Amundsen and Benjamin Young, and this is the third year it has run at Greenville.

An un-conference invites attendees to talk on any subject they choose amongst peers and some of the most experienced people in the community in a very relaxed, informal atmosphere. It provides an opportunity to spend three days immersed in discussions about REST and hear about real world Hypermedia APIs. The idea - everybody talks and everybody listens.



Hack day

Day one was a hack day organised and run by David Zülke. David provided a custom hypermedia type capable of representing a help desk containing users, tickets and comments. He also provided ideas for possible applications that could use the hypermedia type. These ranged from the obvious client / server combination (servers using a custom storage vs servers using an existing ticket storage), to intermediary servers that were capable of enhancing the media type with search capabilities, an automated test client and an XML to JSON proxy.

We split up into teams and 'flew blind' for the first few hours whilst we build up our applications. Once we started getting enough features implemented, we started to be able to interact with each others apps using the common language of the hypermedia type. It was really impressive to see the fruits of the days labours prove to have been worth it - with a quite feature-full help desk application suite having been produced, driven by Hypermedia.

It was particularly pleasing to see a client written in Ruby successfully talking to a server written in PHP using GitHub Issues as a storage medium. The same client could also talk to a second server written on a different platform with its own custom storage engine. A true example of why arguments about language X being better or worse than language Y are left being irrelevant when language neutral constructs are used to bridge the gap between them.

The exercise was a demonstration of the level of interoperability that following the constraints of REST bring to an application system, and the realisation that intermediaries (servers that proxy requests between the client and server end point) can not only cache requests to help an application scale up, but can also enhance the product by decorating the hypermedia type with new link relations and adding extra functionality.

Talks

The following two days were filled with 90 minute workshop sessions and lightning talks. Darrel Miller kicked off day two with his presentation on "The 7 R's of Hypermedia", exploring the practical use of Hypermedia through seven scenarios where it can be used within distributed applications: Relations, Resources, Reference Data, Restriction, Re-Distribution, Reduction, and Reactive. Video.

Pat Cappelaere of the NASA SensorWeb project followed with his work shop on "User Stories and Acceptance Criteria for RESTful Services", demonstrating the RIP (REST Interface Policy) checker, http://rip.jit.su. This is an app that assesses and reports on RESTful services using a continuously updated set of user stories and acceptance criteria to assess the level of compliance of the service against the Richardson maturity model.

The talk and application (RIP) was a great example of the potential of using standard hypermedia types and the benefits of using them in an api over a custom type that only one client is capable of understanding. Video.

Next up was Mike Amundsen, with his amusing run through of the constraints of REST, "REST-like Wisdom from Celebrity Internet Memes". From Inigo Montoya to Yoda, it was a fun walk through of the do's and don'ts of services on the web and provided an entertaining talk to follow lunch. Video.

After Mike was the start of the first session of lightning talks - these run from 5 to 15 minutes in length (the longer ones usually due to active conversation after a short presentation introducing the topic). These give people an opportunity to present anything they want, from a recent Hypermedia project they have been involved in to a monitoring system for performance analysis in front of a targeted audience of REST/Hypermedia professionals, an audience that could only be found at an event like this! Great discussion and everyone who presented could take away a lot from the experience.

I presented some options for how to represent the changing state of resources with Hypermedia. This is a topic that keeps cropping up for me and I solve it in a new way, almost every time I come up against it. The social events with the group once the conference day was done also provided opportunity to discuss the lightning talks and new ideas over a beer in a more relaxed atmosphere.

The day rounded off with a keynote presentation from Stuart Charlton titled 'Linking Data and Actions on the Web', in which Stuart introduced ideas for introducing the concept of Linked Behaviour Trees to the design of REST API's. This was a fascinating exploration of advancing the art of RESTful APIs, and left everyones mind reeling - especially after a full day of other great talks on the subject! Definitely one that I will need to watch for a second time to full absorb what was said and i'm looking forward to seeing the video again once it's made available. Video.

The following day was kicked of by a really great featured talk by Leonard Richardson (creator of the Richardson Maturity Model, which he introduced at a QCon talk in '08), titled 'How to Follow Instructions'. Well worth a watch, even if just for the excerpt on Native Language Bindings; 'Native Language Bindings [for REST APIs] are generally bad. Not like poisonous snakes are bad… but like fast food restaurants in neighbourhoods that don't have grocery stores are bad. They're compensating for a failure earlier in the process. [snip] And it can be really tasty, but just wait six months after you eat one of these every day!'. Very humorous talk from a well respected author and speaker in the industry. Video.

Pat Cappelaere spoke for a second time on 'RESTful Patterns for [OGC] Enterprise Services', a dive into what patterns are used in a network of millions of sensors, both in orbit (satellite) and at ground level, to bring them together and make them accessible to both humans and machines. This followed his earlier talk on RIP, and how the SensorWeb team are using it to drive the definition of the Enterprise Patterns they use today. Video.

Matt Bishop followed Pat with his talk titled 'HATEOAS Your Cake and Eat It Too', a walk through of how Matt and his team at Elastic Path turned a complex e-commerce process into a Hypertext driven API, and how a true Hypermedia driven api (Level 3 of the Richardson Maturity Model) can only be successful if it's based on real world (reality) concepts rather than the existing services that it's built on. He goes through some of the common misconceptions of what constitutes L3 that they have come up against during their journey towards L3, including "L3 is HTTP used correctly" and "Content-Type is the Hypermedia-Type" where he introduces the concept that the Content-Type (xml or json) is actually different from the Hypermedia-Type (atom / hal / etc). One of the most interesting ideas he introduced was the 'Zoom' functionality of the framework they built - a way of expanding out responses with many directly related resources in a single request to avoid making client to have to make many requests to the server, something that the HAL media type specification supports natively through embedded resources. Video.

The final featured talk of the conference was by Sam Ramji of Apigee, on 'Architecting for performance with devices & APIs', with some lessons learned from the history of computing on low bandwidth, high latency networks now that we're moving back towards that world with the popularisation of mobile devices. HTTP APIs are the heart of most applications we run on mobile devices so taking consideration of the network on which the data will be transmitted over is essential to the success of the API project. He introduces the idea that from the perspective of the mobile client, the Internet is a database, and that the API is the modern day equivalent of writing stored procedures on the database to optimise it. Sam included some enlightening tips on how 3G devices work, and how a 'chatty' API can cause a serious drain on battery powered mobile devices. Video.

The remainder of the conference day was filled with lightning talks on a variety of subjects from 'How to be a better Condescender-er' to people introducing new tools they have been developing or working with and new ideas for advancing the art of Restful API design. The lightning talks are all available to watch on the RESTFest vimeo channel.

All in all it was an excellent three days in Greenville. I personally came away with new ideas and thoughts on the 'right' way of doing things when designing Hypermedia systems and had the chance to meet and talk with many of the people I have known through various online forums.

In closing

Thank you to the organisers, Mike Amundsen and Benjamin Young for putting together the event and to the sponsors for making it all possible. The food during the day and social events in the evening were all a great success.

We (Inviqa) are putting together a training course on creating RESTful Hypermedia Application which will become available from December. It will cover Service Oriented Architecture and a recap of HTTP, the concept and constraints of the REST architectural style and how it's evolving, and of course - the role that Hypermedia plays in all of this - and an opportunity to practice these concepts in a training environment. Get in touch if you want to know more.

                          
]]> http://techportal.inviqa.com/2012/10/03/restfest-2012/feed/ 0 http://techportal.inviqa.com/2012/09/07/symfony-live-london-ticket-winner-announced/?utm_source=rss&utm_medium=rss&utm_campaign=symfony-live-london-ticket-winner-announced http://techportal.inviqa.com/2012/09/07/symfony-live-london-ticket-winner-announced/#comments Fri, 07 Sep 2012 14:08:11 +0000 Lorna Mitchell http://techportal.inviqa.com/?p=4254 We are delighted to announce that the competition winner is Manuel! Manuel will win a pass to the Symfony Live conference day next week. We hope you have a good time!

Thanks to everyone who left a comment on the post, we loved hearing your reasons for wanting to attend. If you would like to attend there are still a few places available, so get your tickets and we'll see you there.

To find out more about the event and keep up to date on the day, make sure you're following @SensioLabsUK. We'll be tweeting news, views and pictures throughout the day.

                          
]]> http://techportal.inviqa.com/2012/09/07/symfony-live-london-ticket-winner-announced/feed/ 0 http://techportal.inviqa.com/2012/08/24/symfony-live-london-13th-14th-september/?utm_source=rss&utm_medium=rss&utm_campaign=symfony-live-london-13th-14th-september http://techportal.inviqa.com/2012/08/24/symfony-live-london-13th-14th-september/#comments Fri, 24 Aug 2012 13:31:00 +0000 Lorna Mitchell http://techportal.inviqa.com/?p=4223 The first ever Symfony Live London conference will be taking place next month, hosted by our sister company Sensio Labs UK! The two-day event involves one full day of in-depth training, followed by a 2-track, deeply technical conference.
(We’ve got a ticket to give away, keep reading!)




Workshop Day

The workshop day gives you the choice of two workshops designed to enable you to get the best out of Symfony whether you are an experienced user, or new to the Symfony world.
The first training course, Getting Started with Symfony2 will give you an insightful introduction to the framework, from installation to configuration and request management.
The second option for the workshop day is more advanced - Going Further with Symfony2. This workshop dives straight into some of the more advanced features of Symfony2, such as mastering the dependency injection container configuration, running automated tests and Doctrine integration.

Conference Day

We’re extremely lucky to have two major project leaders as our keynote speakers. The conference day will start with an opening keynote from Dries Buytaert - the Drupal founder and project lead. Then we’ll close the conference with a session from Fabien Potencier, the founder of Symfony and a major contributor to the project.

The main part of the day will consist of two tracks of very technical talks from some well known speakers from the Symfony2, Drupal and PHP conference scenes. There are a variety of topics from key elements of the development process (PHPDocumentor, code reviews, Composer, BDD), through essential skills for using Symfony2 (bundles, SonataAdmin) and including considerations for the platform (ElasticSearch, Amazon Web Services). With all this technical content, the hardest part of the day will be choosing which track to attend.

Symfony Certification

For the first time in the UK, developers will be able to sit the Symfony2 Certification exam.

For organisations using Symfony, having certified developers on your team really shows the world you don’t compromise on quality when delivering projects. Perhaps more importantly, it also shows you work closely with your developers to improve their skills and engage with the Symfony community.

For individuals and freelancers, working to improve your skills to the level required to pass the exam will give you a much more robust knowledge of Symfony as it will encourage you to examine areas of the Symfony toolset you may not have used before. Once you have the acquired the knowledge required passing the exam proves to potential clients and employers that you have worked hard to set yourself out from the crowd.

Distinguish yourself from other Symfony developers and become an official SensioLabs Symfony Certified Developer.

Win a ticket

We have one ticket to give away to our readers. To be in with a chance of winning, simply leave a comment on this post telling us why you think you should win the ticket. Make sure you include your email address so we can get in touch if you win! We will randomly draw a winner when we close the competition on September 5th. Want to join us? Then, leave a comment.

Unfortunately we only have one ticket to give away so if you don’t win we’d still love for you to join us at the event. If you’ve not already signed up tickets are still on sale. Grab one while you can.

To follow the news from the event follow our twitter account @SensioLabsUK or the #symfony_live hashtag.

                          
]]> http://techportal.inviqa.com/2012/08/24/symfony-live-london-13th-14th-september/feed/ 14 http://techportal.inviqa.com/2012/07/31/custom-zend-framework-form-elements/?utm_source=rss&utm_medium=rss&utm_campaign=custom-zend-framework-form-elements http://techportal.inviqa.com/2012/07/31/custom-zend-framework-form-elements/#comments Tue, 31 Jul 2012 09:12:34 +0000 Gavin Corbett http://techportal.inviqa.com/?p=3884 Creating forms in Zend Framework (ZF) is easy, but creating and managing complex forms can get messy! Using config files is perfectly good if you use standard Zend Form elements; it is quick and easy to setup a form in a few minutes however you will have limited control over how the form is rendered. If you want to display your own custom form element correctly in Zend Framework then some configuration is needed in order to achieve this with a config file. An alternative approach is to create some custom objects that extend the base Zend Form object - you can still use the config but also add in your own custom Zend Form elements.

If you use a custom form object then all of your form logic can be grouped together. The object can include data validation with Zend validator (and/or normal PHP validation for more complex validation routines), form data retrieval, form view format and element view rendering. This has the combined benefit of freeing the controller of any form logic needed for the custom objects and ensures the "skinny controller, fat model" approach can be used, avoiding duplicating logic as each form object is now responsible for its own data.

This article will show you how to create custom form element objects which change the standard view template for an element - the code from all the examples is available on github too, if you want to follow along. The approach shown here enables us to add logic into the element, so that the element behaves the same way in any form. A good example could be a select box that can pre-fill data from a database table and automatically add a Zend Validator to the array. We'll also cover how you can have total control over the way the form is rendered, by using a view script. As an example, we'll create an e-commerce checkout where a billing address and delivery address contain exactly the same fields with JavaScript and Zend Validation; our postcode fields will trigger a postcode lookup.

Creating a Custom Zend Form Object

Extending the Zend Form the Right Way

To gain great functionality on top of ZF, the best way is to extend the framework, creating your own base, then extend that per project, as shown in the code example below. This way if you want to add a new method to an object then you just need to add it to your own main object, leaving the core Zend object unmodified. For instance if you want every Zend Form to have a getValidatedFormData() method then you can implement it in your base class (and should Zend implement the same functionality later on then you can remove your implementation, or leave it and override theirs). Hopefully this seems obvious but I am amazed how often I see code that uses only the core Zend objects.

class MyBase_Form extends Zend_Form
{
....
}


Your main base class doesn’t need to do any more than just extend the base Zend Form object; this makes it easier to update all your objects with new methods or override the core Zend methods should you need to do so in the future. The specific form (in this case Email_Form) should extend your base Zend Form object we just created, so that any and project specific-methods can be added leaving you to re-use your MyBase object.

This level of abstraction may seem slightly "over the top", however it can save you a lot of time later if more changes are needed such as changing the default display from "dt" and "dd" tags to a table layout. With the abstraction layer, this only needs to be done once and all form objects will be converted. This approach has the added benefit of easier testing as you only need to test the functionality you are actually writing.

So far, all we have done is set up a form object, now we can configure the form’s elements and more importantly the validation. Using this approach we keep the validation logic together and, should the elements change, the validation can change to match. The changes are unit testable and re-useable, and validation will be also available if this form is to be used as a sub-form.

Adding the Elements

To add elements to this form, we could use a Zend_Config objects for adding standard elements only, but we get more control by adding the elements manually - so we'll do that!

class MyProject_Form_Address extends MyBase_Form
{
protected $_action_url = '/[controller]/[action]/';

public function init()
{
parent::init();
$this->setAction($this->action_url)
->setAttrib(‘id’, ‘project_form’)
->setMethod(‘post’);

$billing = new Zend_Form_Element_Textarea('billing_address', array(
'label' => 'Billing Address:',
));
$billing->setRequired();

$del_diff = new Zend_Form_Element_Checkbox('different_delivery_address', array(
'label' => 'Deliver to different address:',
));

$delivery = new Zend_Form_Element_Textarea('delivery_address', array(
'label' => 'Delivery Address:',
));

$submit = new Zend_Form_Element_Submit('submit', array(
'label' => 'Send:',
));

$this->addElement($billing, 'billing_address')
->addElement($del_diff, 'different_delivery_address')
->addElement($delivery, 'delivery_address')
->addElement($submit, 'submit');
}
}


As you can see from the above example, we have used standard objects but we have added attributes to specific elements in the form; this can be further extended by using the decorators (something we will cover later in this tutorial).



Validating the Form

For validation what place is better to check whether this form is valid than the form itself?  If a field is required only when a certain condition is true, you could check this every time in your controller. Alternatively you could extend the isValid() functionality and use any properly-extended Zend_Validation_Abstract object on any of the form elements.


public function isValid($value, $context = null)
{    
    $valid = parent::isValid($value, $context);
    if ($this->isDeliveryAddressDifferent()) {
        $this->getElement('delivery_address')->setRequired();
        return parent::isValid($value, $context);
    }
    return $valid;
}


As you can see the isValid function now checks to see if the form element is checked and then set's the delivery address to be required and revalidates the form, this way the form is more dynamic.

Getting the Data from the Object

There is the standard way of getting data from the object by getting the raw data array by using the getValues(), however there is a better way: create custom "getters". By using getters, you can add more logic if it is needed, but the biggest benefit is that it will make your code much more readable - and of course again unit testing will be easier.

As you can see this way of getting the inputted data is simple to do, and it will make your code much more extendable and maintainable.

public function isDeliveryAddressDifferent()
{
    if ($this->getValue('different_delivery_address') == 1) {
        return true;
    }
    return false;
}

public function getAddressLine1()
{
    return $this->getValue('line_1');
}

Changing the Way the Form is Rendered

As with most of Zend Framework, you can use as much or little of the standard framework as you like. In particular let's look at ways of changing how the form is rendered, beyond simply adding a stylesheet. Sometimes you need something more than the standard "dd" and "dt" output, perhaps to output as a table or have full control over the view template for a given form or sub-form. This can be done by adding to the init() method.

public function init()
{
    $this->setDisableLoadDefaultDecorators(true);
    $this->setDecorators(array(
        array('ViewScript', 
            array('viewScript' => 'form/_form.phtml')),
            'Form'
    ));
}

The full path for this script would be application/views/scripts/form/_form.phtml

Creating a Custom Form Element

Extending Zend_Form objects is good practice but there is very little that you can’t do with Zend_Config. However when you come to add your own custom elements, this approach will really pay dividends. So now I will show you how to extend the standard form elements to make your own, such as creating a select / radio / checkbox set of elements from a database resultset, or using a Javascript library to create a date time field with custom view helpers, adding standard Zend_Validtors that it has to pass before the element is valid. In this tutorial, I will be extending a text box to make a standard email textbox, and pulling data from a database to pre-populate a select box.

Extending the Zend Form Element Object

Take a look at the following example; we simply add two validators when we initialise the object:

class MyBase_Form_Element_Email extends Zend_Form_Element_Text
{
    public function init()
    {
        $this->addValidator('EmailAddress')
             ->addFilter('StringToLower');

        return parent::init();
    }
}

Now we've done this, those validators are automatically added everywhere this element is used. This means that your form needs to do no more than this: when you create the form, just give the name of the element you created rather than the standard text field.

Connecting to a Database for Pre-populated Elements.

The example below might look a little more complicated, but we will approach it step by step. The hardest part is getting the data from the database, which shouldn't be too tricky (you could refactor this so that the data is pulled from a data object however for this tutorial I will just use the standard Zend_Db_Adapter). Here we are creating a select element but, by changing the parent object, it could equally be a set of radio options form or even a set of checkboxes instead. N.B. There is a obvious SQL injection issue here, as you know you should validate ALL input, for simplicity I have not added this but I do recommend that you implement something on all SQL variables. Here's the code example first, then we'll step through each bit below:

class MyBase_Form_Element_SelectFromDB extends Zend_Form_Element_Select
{
    // First element in the select box
    protected $_select = array(
        '=== Please Select ===',
    );

    public function __construct($spec, $options)
    {
        parent::__construct($spec, $options);

        $this->addOptionsFromDatabase(Zend_Registry::get('db'), $options)
             ->setRegisterInArrayValidator(true);
    }

    public function addOptionsFromDatabase(Zend_Db_Adapter_Abstract $db, $options)
    {
        $table = $options['table'];
        $columns = $options['columns'];

        $sql = sprintf('SELECT `%s` as id, `%s` as name FROM `%s`', $columns[0], $columns[1], $table);

        $row_data = $db->fetchAll($sql . ';');

        foreach ($row_data as $row) {
            $this->_select[$row['id']] = sprintf($this->_pattern, $row['name'], $row['id']);
        }
        $this->addMultiOptions($this->_select);

        return $this;
    }
}

First, take a look at the constructor. We try and make the element as flexible as possible and as we don't know which columns, table or even database this element would be communicating with, we need a way of getting that from the calling object. As in all Zend_Form_Elements there already is a second parameter which does different things depending on element, so we will utilise this and append our details to this array parameter, specifically for the database. In this example I use a rudimentary array but you can always refactor this to take an object, for example.

As part of the initial object construction, we populate the Zend_Form_Element_Select option array from the database; this also adds the Zend_Validate_InArray to prevent undesired details being entered into the form.

Next we shall look at getting the data from the database. This method mostly constructs the valid SQL, which I assume you are familiar with. This method loops through all of the database rows, then converts from a multi-dimensional array into a single-dimension array with the key from the database as the index. The text will be displayed in the form that the element has rendered.

Creating a Custom View Helper

There are times when you would like to use a standard element but with a custom view, e.g. a postcode lookup with javascript validation, or a date field. I will show you a quick way of doing javascript validation that will accept any input and add an OK button next to the element. There are two files that need to be created: a form object which extends the standard element, and the view helper.

class MyBase_Form_Element_Email extends Zend_Form_Element_Text
{
    public $helper = 'formEmail';
}

As you can see this object doesn't need to do any more than extend the standard object and change the view helper.

Now lets create the view helper: It is important to notice that the filename is FormEmail.php and the method name is formEmail – this is taken from the $helper. So Form{$helper}.php and form{$helper}.

class MyBase_View_Helper_FormEmail extends Zend_View_Helper_FormText
{
    public function formEmail($name, $value = null, $attribs = null)
    {
        $info = $this->_getInfo($name, $value, $attribs);
        extract($info);

        $xhtml = $this->formText($name, $value, $attribs) .
                 'view->escape($id) . '" style="display:none;">OK';

        $xhtml .= '
        
             $("#' . $this->view->escape($id) . '").blur(function() {
                if($("#' . $this->view->escape($id) . '").val() != "") {
                    $("span.' . $this->view->escape($id) . '").show();
                } else {
                    $("span.' . $this->view->escape($id) . '").hide();
                }
            });
        ';

        return $xhtml;
    }
}

This example is fairly simple and you would refactor to move as much logic as you could into the CSS, and create a JS with a function call taking in the id from the form element (which should be unique).



As you can see, the approach outlined here makes your controllers very easy to maintain, but more importantly the objects that know the most about the input from the user now contain the validation and filter logic - and so it can be reused and extended easily. Check out the github repository which you can find here ZF1-CustomElements, this is a fully working example of the above code, which also includes the pre-filling of the select from a database table. Hopefully this has given you some ideas for creating more manageable forms in your ZF1 projects, do let me know in the comments what you think!

                          
]]> http://techportal.inviqa.com/2012/07/31/custom-zend-framework-form-elements/feed/ 0 http://techportal.inviqa.com/2012/07/09/do-you-need-a-multiregion-website-deployment/?utm_source=rss&utm_medium=rss&utm_campaign=do-you-need-a-multiregion-website-deployment http://techportal.inviqa.com/2012/07/09/do-you-need-a-multiregion-website-deployment/#comments Mon, 09 Jul 2012 08:18:52 +0000 Adam Smith http://techportal.inviqa.com/?p=3819 In this age of cloud computing, it has become far easier to choose, or even move, the geographical location of your web server. It has also brought high-availability solutions such as hosting simultaneously in multiple locations into the reach of a even very modest applications. But where is the best place to host your site? And are the improvements worth the effort of distributing your servers across the world?

Armed with a few simple tools, you can come a long way to answering these questions fairly quickly.

To begin, let's tackle the first question first. Where is the best place to host a site? As with most things, the answer is "it depends". In the simplest case, all of your site's users are geographically close to one another, so you should host your site as close to them as possible. There are a number of anecdotal examples that show reducing load times for a web page can improve user satisfaction and even sales conversion rates. Thanks to Canadian company Strange Loop Networks, some of the key headlines from these reports have been collated into a single poster. So if you are lucky enough to have all your users together and you're not hosting close to them, do it now for a quick win.

If your site doesn't fall into the "all my users are in the same place" category, the best way to find out where to host is to benchmark. My tool of choice for this is the API provided at webpagetest.org. This is an open source project, so you can download the source and set up your own version, but there is also a hosted version available to use for free, just with limits on the number of requests one person can make per day. In order to use the hosted version you must first contact the site to request an access key, but once you've acquired that, the API itself is extremely simple to interact with.

Preparing the Setup

In order to run speed tests, you first need a test server setup. When running through this process myself I was trying to establish a good place to host servers for the UN's World Food Programme. Inviqa have been tasked with re-architecting the API that underlies their online fundraising efforts. The API must be responsive globally, but there is a particular focus on keeping the administration tasks performed in Asia quick.

Four potential server locations were identified, each was one of Amazon's EC2 regions. A server instance was booted in each of the four locations using a standard EC2 image. No modifications were made to the server, other than to place an identical 7KB PNG image, taken from one of the client's existing websites, into a web accessible folder. Tests were then run, in the manner outlined below, against each of these four instances from nine locations around the world.



the image used for testing

You can find the code for all the code examples in this article on github at: https://github.com/ams2435/WebTest

Running the Tests

Using functions built directly in to PHP, we can communicate with the API in a very simple manner. The file_get_contents() function is capable of reading directly from a URL and the API we're interacting with takes all of its parameters from the query string, so we can grab data with just one function call. This is a quick and dirty solution to the problem, but it works well for this example.

The parameters we will pass to the API are:

Parameter	Description	Value used
f	The format of the response	xml
k	The access key for the API	The access key we obtained to access the API
fvonly	Tell the API whether we only require a “first view” of the test URL. The API can run a “second view” of the same URL for each test, which will show us the load times after any browser caching has taken place	1, for these tests we are not interested in finding out the speed of the cached version
runs	How many times to repeat the test	5
location	The location and speed profile for the test runs	Various locations, each with a DSL speed profile – the equivalent of using a 1.5Mbps broadband connection. The full list of locations can be obtained at http://www.webpagetest.org/getLocations.php
url	The URL to test	Various URLs representing each test server

Since we are requesting our result in XML format, we will also want to convert the returned string using the simplexml_load_string() function. Putting this together gives us the following code snippet:

protected function _runLocation($url, $location){ $xml = simplexml_load_string( file_get_contents( 'http://www.webpagetest.org/runtest.php?' .'f=xml' .'&fvonly=1' .'&runs=5' .'&location='.$location.'.DSL' .'&k=PERSONAL-API-KEY' .'&url='.$url ) ); if($xml->statusCode == '200'){ return $xml->data->testId; } }

You can see from the above that we are sending off the request to the API and receiving a testId in response. This is due to the asynchronous nature of the web test API; the test will not have actually been completed yet. We can use the returned testId to ask for the results once the test has completed, so we need to store this somewhere so that we know what results we still need to gather.

You may already have noticed that the above code is wrapped in a protected function, this function is called from the following piece of code:

public function run(){ $db = new WebTest_Database_Connection(); foreach($this->_locations as $location){ foreach($this->_urls as $url){ $testId = $this->_runLocation($url, $location); $db->query('INSERT INTO Runs VALUES ("'.$testId.'", "'.$url.'", "'.$location.'", "'.date('Y-m-d G:i:s').'", 0) '); } } }

We are simply storing the IDs returned from the _runLocation() function into a database, along with the URL, location and date of the run. You can see from the code that we're running through two nested foreach statements so we can perform the tests in a number of location/URL combinations and then compare the results. The two functions shown are simply wrapped up in a WebTest_Runner class that also has an addLocation() function which stores each location that is passed to it for use in the above loop. The constructor to the class also takes an array of URLs to test. This leaves the triggering of tests looking as simple as this:

$runner = new WebTest_Runner( array('http://test-url-1.com', 'http://test-url-2.com') );   $runner->addLocation('Korea'); $runner->addLocation('China'); $runner->addLocation('Australia');   $runner->run();

The code shown here triggers six tests using the web test API, one for each of the URLs in each of the locations specified. We will then have a testId in our database for each of these tests. Remember though that we have asked for five runs for each test, so our six test ids will actually relate to 30 individual speed tests.

Gathering the Results

The next thing we have to do is obtain those test results now the tests have completed. This is how we can do that:

$result = simplexml_load_string( file_get_contents( http://www.webpagetest.org/xmlResult/'.$testId.'/') );

This will return us either the test results we have asked for, or if the results are not yet available, an error code. We can continue to poll the API for these results as many times as we like, but in order to perform any analysis of the results it is better to store them in our own database. The following function does just that:

public static function gatherResults(){ $db = new WebTest_Database_Connection(); $runs = $db->query('SELECT * FROM Runs WHERE ResultsGathered = 0'); foreach($runs->fetchAllRows() as $runRow){ $run = new WebTest_Result($runRow['TestId']); $loads = $run->getAllLoads(); if($loads){ foreach($loads as $runNumber => $load){ $db->query('INSERT INTO Results VALUES ( "'.$runRow['TestId'].'", '.$runNumber.', "'.$run->getUrl().'", "'.$run->getLocation().'", "'.date('Y-m-d G:i:s',$load->date).'", '.$load->loadTime. ')' ); } $db->query('UPDATE Runs SET ResultsGathered = 1 WHERE TestId = "'.$runRow['TestId'].'"'); } } }

This function simply loops through all the existing testIDs we have not yet collected results for, and fetches the results. It then stores those results in a local database, looping through each of the five runs per test id and storing them all individually. Finally this function marks each result set we successfully download and process as having had its results gathered. The API access shown in the previous code snippet is happening here inside the WebTest_Result class.

Once this process has been repeated a number of times, you will have built up enough test data to be able to start analysing it. As the data is stored in an SQL database, we have the full power of SQL to analyse the results. Making use of the MIN(), MAX() and AVG() functions of SQL will give results that should begin to show a picture of how your site is responding from different locations.

The results below are those gathered by Inviqa for the re-architecting project.

The times shown are the full browser load time for the image, including DNS time, connection time, data transfer time and rendering time. This is clearly not a real world test, but since we have the same simple setup in each location, it is sufficient when simply comparing the different locations. The results were collected at intervals over a three-day period and average load times in milliseconds were recorded for each location.



Server response times by location (click for a bigger version)

	Tokyo Server	US West (Northern California) Server	EU West Server	US East Server
Sydney, Australia	600	855	1217	981
Jiangsu, China	1124	1536	3634	1888
Geneva, Switzerland	1213	896	537	692
Seoul, South Korea	736	802	1223	1056
London, England	1038	725	454	598
Los Angeles, USA	757	450	905	602
New York, USA	874	537	601	329
Sao Paolo, Brazil	1265	989	1036	749
Singapore	580	894	1442	1148

The first thing to notice is that the results back up our basic assumption that you should host as close to your users as possible. It is clear that should all of your users be in Singapore, the best of the tested locations is the closest (Tokyo). However, it isn't always as simple as the actual distance, due to the way the data trunk cables are laid out across the world. Straight line distance between Seoul, South Korea and the EU West data centre in Ireland is approximately 9,000km, the distance from Seoul to the US East data centre in Virginia is in excess of 11,000km, however, as the results above show, the latency to US East from Seoul is actually lower than that to EU West. If you wish to delve deeper into the trunk cable locations, a map can be seen at: http://www.submarinecablemap.com/. The map shows you the different potential routes data can take across oceans, something I have found interesting to look at on more than one occasion.

Selecting a Server Setup

Once you have collected comparable results for the locations you're interested in, it's time to select a server setup.

Sticking with a Single Location

The simplest solution is to pick a single location that offers a good speed compromise for all your users. You can see from the results that in some cases the additional expense and complexity of geographically distributing your application would bring a speed increase of fractions of a second, which isn't much of a return on your investment. I would suggest that if you're targeting users in London and Los Angeles, rather than hosting in both EU West and US West, taking a 150ms hit from both locations and hosting in US East is a better option. After all, if you find once the application is up and running that 150ms is a deal breaker, you can always move to a more distributed approach at that point. This approach may not follow the Google advice that every millisecond counts, but not everyone has the resources that Google have to implement complex and expensive solutions, and there are other effective ways to speed up a site before resorting to a multi-region setup.

Making it More Robust

Despite picking one location, you don't have to put all your server eggs in one data centre basket. If you decide to go with one of the Amazon regions, you will find they are all split into “Availability Zones”. An Availability Zone within a region is a completely separate data centre; Amazon offers a load-balancing service which will balance your load across these two data centres. Then if one Availability Zone becomes unavailable, all of your load will transparently go to the other zone, making it very easy to create a highly-available solution with a level of redundancy that is far more difficult and expensive to achieve without a cloud hosting provider. In order to distribute an application across different Availability Zones, it is usually necessary to do some data replication. Within each region, all of Amazon's Availability Zones have high-speed connections to each other, meaning that doing this kind of data replication quickly is entirely possible. The diagram below shows a potential server setup across two availability zones. You can see from the diagram that the web servers should make all of their database writes to a single master in one of the zones, however, all the reads will be done from a database in the same availability zone.



Example server setup across Availability Zones

In this setup, if Availability Zone 2 becomes unavailable, nothing needs to be done for the application to continue running. If Availability Zone 1 becomes unavailable, the database slave must be promoted to be the master, but then the rest of the setup will continue working as expected. If you're using both a MySQL database and EC2, you might even want to look at the Amazon RDS offering. RDS can do this data replication across availability zones for you with virtually zero maintenance effort, including automatic slave promotion and recovery.

Utilising a Content Delivery Network

No matter where you decide to locate your servers, there are other tricks you can employ to improve page load times. A good place to start is to employ a content delivery network (CDN). This is essentially distributing across many different locations, but only static resources are distributed. It is very often the case that the slowest part of a page load is receiving all the images, JavaScript and CSS files associated with the page. By adding these resources to a CDN, the user will load them from the nearest location to them and see a performance increase, but you haven't had to do anything complicated to make the application truly multi-region. There are many different providers offering CDN solutions, they work by using a DNS trick to route the client to the fastest responding location for a domain name. This means you could set “images.mydomain.com” as the root address of your CDN, then reference all the images in your site with that subdomain; they will then always load from the fastest location for each user.

In addition to employing a CDN for your own static resources, if the resources are common resources, for example the base jquery.js file, loading it from a public CDN will offer the potential benefit of a user already having a cached copy before theyvisit your site for the first time – this really is a free performance boost as it's difficult to ask for better than a locally-cached copy of a resource in terms of speed. This works due to many different sites using the exact same URL for the resource, so as far as the browser is concerned it has already loaded it. Google's public libraries are very popular for this, a full list of those available can be seen at https://developers.google.com/speed/libraries/devguide.

Full Multi-region Implementation

If you have a compelling reason to go for a full multi-region setup, it is possible to use the same principles that CDNs employ to direct your users to the nearest server setup you have. The technique is called “latency-based routing”. There are a number of providers of such routing, and a recent entry into the market by Amazon, as part of its Route 53 DNS offering, is likely to make the practice more common. The advantage of using Amazon's Route 53 service for this is how simple it is to integrate with the other Amazon services. To implement latency based routing, you set your application up in as many different locations as you require and then tell the routing service where all of your locations are. When a user hits your master URL, the location that will serve the user the quickest is automatically calculated, and the user is directed to that location.



Users are directed to the quickest location when employing latency-based routing with servers distributed around the world

The real difficulty with setting up your application this way is trying to share the data between all the locations – it is not simply enough to assume that a user will always hit the same location. Even if your locations never go down, someone might need to get at their data while they are travelling, or something could happen to a key piece of infrastructure that suddenly makes a different location respond quicker to a user. You can see from the diagram above just how complicated the synchronisation of data could be. For the four server locations that have been selected, represented by large dots on the diagram, six different synchronisation paths exist. It is this synchronisation of data that makes full multi-region setups incredibly complex. While it is possible to set up a master-slave replication in the same way we would to have the application work across multiple availability zones, the network latency is likely to be far too much of a limiting factor; any speedup that had been achieved by hosting multi-region will be lost. It is therefore necessary to utilise a master-master replication policy, which tends to require a much greater level of maintenance and is very easy to get wrong!

Keeping a Master Region

Maintaing a master region is a less daunting route to creating a multi-region setup. This is very much like having a single location setup, but with some extra server power making sure that the really important transactions are kept speedy. This approach relies heavily on having an asynchronous architecture within your application. The key here is making it appear to the end user as though the actions have been completed quickly, even if they haven't yet been completed. Rather than waiting for a set of database queries to run before returning a response to the user, respond immediately to say the request is being handled. You can then push the request onto a queue, which the master location will pick up and process. While this doesn't eliminate the need to communicate across the world on each request, the user is not waiting for that communication to take place, thus keeping the experience fast and keeping satisfaction high.

A major drawback of this solution is ending up with an inconsistent experience for users. While most requests can easily be queued for actioning later, some things cannot wait. If a user wants to pay for a product, we can't ask them to wait a few minutes while we catch up before taking the credit card details! This means that these immediately-required processes will have slower response times than the rest of the application, due to having to make the user wait for the requests to travel to the master region. Still, with some clever usage of caching, this solution can be really attractive. If you truly have a requirement for global distribution of servers, this option strikes a nice compromise between complexity and performance.

Conclusion

We began our journey with two questions to answer. Hopefully I've been able to show you that picking a location to host your site is fairly simple once you know where your users are. The thing I'd most like you to take away from reading this article is that selecting the location for your server is an important decision. There is a vast range of different providers and locations that you could pick, and selecting the right one could make a huge difference to your end users.

Is it worth distributing your servers across the world? I think in most cases the answer to this question is no. It's important to make your site work as fast as possible; sales can be increased and users made happy by even modest improvements to response times. However, the added complexity and cost of distributing your site across the world must be considered carefully. If you will be selling a product specifically to customers in the UK, do you need the website that sells it to run just as quickly for users in Australia? If you have a compelling case for a multi-region deployment, keeping a single master region is my advice. Slave front-ends in locations that need speeding up and a content delivery network to distribute your static content will keep your users happy most of the time.

Finally, if you've never run a speed test against your site using webpagetest.org or a similar service, I urge you to do it at least once. I've only really talked about the page load times reported by the API here, but the web interface is excellent for running one-off tests. The wealth of information and recommendation the site is able to offer is well worth the couple of minutes it takes to type your site's URL into a text box and wait for the test to run.

Code Samples

Remember that all the code examples in this article are available on github at: https://github.com/ams2435/WebTest

                          
]]> http://techportal.inviqa.com/2012/07/09/do-you-need-a-multiregion-website-deployment/feed/ 0 http://techportal.inviqa.com/2012/06/28/conference-report-mongodb-uk/?utm_source=rss&utm_medium=rss&utm_campaign=conference-report-mongodb-uk http://techportal.inviqa.com/2012/06/28/conference-report-mongodb-uk/#comments Thu, 28 Jun 2012 13:15:22 +0000 John Cleary http://techportal.inviqa.com/?p=3847 The 4th annual UK conference for MongoDB developers was hosted by 10gen at the Mermaid Conference Centre in London's Blackfriars. With four tracks running throughout the day - both technical and non-technical, in addition to an "ask the experts" drop-in track, choosing which talks to visit was always going to be a challenge!


Keynote

Eliot Horowitz, Co-Founder and CTO 10gen

The Keynote from Horowitz was excellent and pitched just right. There were around 530 delegates at the conferences this year, and with MongoDB still being a relatively new tool, many will have been attending for the first time (myself included). The talk covered the history and rapid growth of MongoDB as a database as well as 10gen as a company. The company was set up back in 2007 by Eliot and Dwight, and for the first 2 years it was just these two guys. Four years later 10gen now has 140 staff - 17 in Europe alone (most of whom appeared to be at the conference!)

Eliot also covered some of the key design considerations behind MongoDB. These included being easy to setup, easy to scale, and doing the right thing right out of the box. Having used MongoDB in a production environment I can testify that these claims are true.

Schema Design by Example

Christian Kvalheim, 10gen

Christian gave a useful account of the importance of schema design with reference to performance and scalability, the detail of which is beyond the scope of this article. For me, however, two key points came out of this talk.

The first of these is that although it's important to have a good design for performance reasons, it is possible (and in principle straight-forward) to change the design over time. The speaker gave some very good examples of this.

The second interesting point is that noSQL Design Patterns are emerging, which can only be a good thing. With new technology such as this, it's often tempting to wade in and start playing around with the fun stuff. However, as professionals we should always be looking for reusable solutions. I have no doubt that the blogosphere will see articles about noSQL Design Patterns in the months ahead.

MongoDB on Innovation Projects / Products

Noberto Leite, Telefonica R&D

It is refreshing to know that not all the talks at this conference were technical. As much as I love geekery, it's also great to hear how other companies are using MongoDB and what they are using it for. Although Telefonica have only been using MongoDB for little over a year, Noberto was able to give plenty of examples of the kinds of projects they had undertaken. Of particular interest was a project which had originally been developed using Oracle at the back end, and had taken two years to complete. The system was slow and deemed not fit for purpose so it had to be rebuilt from scratch, this time using MongoDB. The new system was around 15 times faster and only took 3 months to build. Definitely a win for MongoDB!

MongoDB for Online Advertising at AOL

Jonathan Reed, AOL

Back in the main auditorium, Jonathan Reed from AOL explained how they built an analytics engine for their online advertising arm Advertising.com. They store huge amounts of global user-tracking data across multiple regions. Interestingly they employed a clever sharding tactic whereby the data was sharded by region, but replicated to the neighbouring regions. This approach allowed for very fast local writes, but still gave the failover if the regional node was unavailable. Overall this was a very interesting talk that demonstrated the power and flexibility of MongoDB.

Ask the Experts

During the lunch break I spoke to a couple of the exhibitors including Cloud Foundry and Equal Experts. 10gen were also offering an "Ask the Experts" drop-in session where you can....well...ask the experts! I grabbed the opportunity to ask a few high-level questions about MongoDB. I was interested to know what the typical use cases are for MongoDB. I got the usual response of archiving, logging, high volume, high availability but interestingly they also came back with "any application where you may need to quickly change the type of data you store". Hmmmm - isn't that anything built using Agile techniques?

This led to my next question – what percentage of new apps being built today should be considering MongoDB for some or all of their data persistence? In my head I was expecting to hear a figure like 10%, but the response surprised me. Although not committing to a figure, when I suggested 50% they didn't flinch. As the day went on I became less and less surprised by that figure.



The New Aggregation Framework

Ross Lawley, 10gen

One of the most exciting additions for developers in the upcoming 2.2 release is the new Aggregation Framework. Once you've seen it you'll understand why. Not only is it around 25 times faster than the equivalent map-reduce functions, but you can run multiple aggregations simultaneously. If you've ever experienced the pain of trying to craft a map-reduce function, you'll be pleased to know that the aggregation framework is very simple to use. Ross ran an excellent talk taking us from the simplest $count and $sum functions, right through to document-specific functions such as $unwind. The real beauty of the framework is how functions can be chained together in what are known as a pipelines, similar to the way you might perform sub-selects in SQL.

I spoke to Ross briefly afterwards and his belief was that the aggregation framework would eventually replace map-reduce altogether due to its flexibility, speed and ease of use.

Six Months of MongoDB at Nokia Bristol

Tom Coupland, Nokia

Another non-technical talk (I told you there were a few) catalogued the experience of one team's adoption of MongoDB. The background to this story was that the team were spending too many of their days dealing with ORM-related issues and had come to the conclusion that far too much of their time was spent doing things other than delivering features. They assembled a crack team of "devops" people to get buy-in and build knowledge across departments from the very beginning. Furthermore, they threw all of these interested parties into the same room, hooked one computer up to a projector and designed the architecture and schemas as a team! What an amazing way to get everyone up-to-speed and invested at the same time.

Why the Schema Diagram is Ruining Enterprise Computing

Sean Reilly, Equal Experts

Having spoken to Sean earlier in the day, I decide to go along to this talk to see what it was all about. Sean is a very entertaining speaker and knows how to convey his message. The gist of the talk was that if you have a schema diagram there are 3 things you should not do with it.

1. Mistake a schema diagram (which is a model of the persistence layer) for a model of the application.
2. Share the schema diagram with anyone outside your team.
3. Put it on a wiki where it can be seen by everyone.

Why not? Because as soon as you do that, someone will login to your database and read your data. This all sounds reasonable ... until one day you decide to make a tiny change to your data model. It's at this point you will be told that you can't because someone else is reading from it and it will break everything! Sounds like a nightmare, but the solution is thankfully rather simple. The answer is force everyone to use an API to access your data. At one point Sean got the whole room to chant "The schema diagram is an implementation detail which you do not need to know". Brilliant!

Sean also produced two marvellous text-to-speech videos that had the whole room in stitches. They depicted the "without an API" and "with an API" scenarios. As soon as these videos are available online I'll post a link in the comments below.

Mobilize Your MongoDB! Developing iPhone and Android Apps in the Cloud

Marek Jelen, OpenShift

These days it seems as if every man and his dog are offering cloud hosting and/or PaaS - and Red Hat are no exception. Marek bravely did a live demo of building an Android app powered by a Ruby back-end all hosted on the OpenShift platform. Barring a couple of minor DNS gremlins, Marek got the whole platform up and running during the talk. OpenShift is a pretty sweet platform and with the backing of Red Hat, its bound to make a big splash in the PaaS pool.

Sharding Best Practices

Daniel Roberts, 10gen

What was perhaps surprising about this talk is that Daniel opened with a list of alternatives to sharding. The logic being that although sharding is amazing and offers huge potential for linear scaling, it also adds complexity to your architecture, and to some extent your code. If you can solve your performance issues by either a redesign of your schema, by adding an index or through buying better hardware, you can at the very least postpone the need to shard your collection.

Following on from that, the speaker looked at some of the best practices in sharding (the clue was in the title I guess), including the idea of choosing a sharding key based on region to give local writes, which can greatly improve performance. If you were just on the brink of sharding some or all of your database, then this talk would have been perfect for you. With a talk of this nature it could have been tricky to hold the audiences' attention for a full 40 minutes, but Daniel's deep understanding of the topic coupled with his accessible presentation style made this talk whiz by.

Thank You and Road Map

Eliot Horowitz, Co-Founder and CTO 10gen

Eliot wrapped up the day whilst holding a pint in his hand. He set the tone for the evening ahead (a five bar pub crawl). As Eliot put it "I'm having a drink now so you can too". Indeed many delegates did listen to the wrap-up whilst sipping beer .

The CTO of 10gen reminded us that although they are all still hard at work fixing bugs and adding features, we should still continue to hassle them with request for the stuff we wanted. The next release, due any day now, will include the aggregation framework, time-to-live collections (where documents can be told when to expire - great for caches et al), in addition to some pretty clever internal stuff that will improve concurrency, reduce fragmentation and generally make everything faster – wow!

On the shortlist for future releases include the long-awaited full text search, more concurrency, online compaction, internal compression and some more security options such as LDAP auditing.

Summary

This was my first MongoDB conference – in fact it was my first database conference full stop. To some extent I did not know what to expect. Actually that's a complete lie! I totally expected back-to-back technical talks and hands on sessions. And if that's what you go to a MongoDb conference for then you wouldn't have been disappointed. However, there was much more besides.

Something else that was mentioned by many of the speakers (no, not just the ones from 10gen) was how great the documentation and support were - both the paid-for support from 10gen and the active community blogs and forums.

The opening and closing talks by Eliot Horowitz showed us how far we've come and where we're heading. Although MongoDB has been around for a few years now, the first release that was considered production-ready only came out in April 2011. Looking at the “Who's Who” of MongoDB users; AOL, Nokia, MTV, Foursquare – the list goes on, on thing is for sure - it's is not a fad. MongoDB is here to stay!

                          
]]> http://techportal.inviqa.com/2012/06/28/conference-report-mongodb-uk/feed/ 1 http://techportal.inviqa.com/2012/06/26/tips-for-vagrant-and-chef/?utm_source=rss&utm_medium=rss&utm_campaign=tips-for-vagrant-and-chef http://techportal.inviqa.com/2012/06/26/tips-for-vagrant-and-chef/#comments Tue, 26 Jun 2012 07:30:26 +0000 Nick Peirson http://techportal.inviqa.com/?p=3807 For those of you cooking up a storm with Vagrant and Chef, these tips might help when creating your VMs (Virtual Machines). These are things that we've discovered and found useful while working on projects at Inviqa. If we thought they were useful, we thought you might too!

Keep Your Base Boxes Up To Date

If you're working on a long-lived project using Vagrant boxes, chances are you set up your base box, spent some time getting it configured how you liked, then forgot about it. If part of your provisioning process is to bring packages up to date, then the longer your project runs, chances are the longer this step takes, due to getting more updates. If you rarely rebuild your box you probably won't notice this, but if you regularly 'destroy' and 'up your box, this can be reduced by updating your base box to include these updates. New developers to your project will also thank you for not spending hours watching updates scroll past, or maybe they won't - if they were hoping for an extra coffee break ;)

To take this even further, you could use VeeWee to build a custom box with all the packages pre-installed, further shortening the provisioning step.

Vagrantfile - not just static configuration!

One realisation we came to on a couple of projects independently, and have since discussed and refined, is the the Vagrantfile is executed rather than just read. This happens early in the start up process, so it's a quick way to check that resources are available rather than waiting for half an hour while your VM builds and provisions, only to find that you forgot to place your custom widget config in your dooberry directory and, apart from the tweets you read in the lull, you've just lost the last half an hour.

This is very useful where you have multiple VMs on a project and want to share config between them without duplicating the config in each block:

Vagrant::Config.run do |config| if ENV['UNIQUE_KEY'].nil? then puts "The UNIQUE_KEY environment variable must be set to a unique value within the project, i.e. your username" exit end   nfs_status = (ENV['WINDOWS']) ? false : true   #Share the code from host config.vm.share_folder "Sites", "/mnt/Sites", "#{ENV['HOME']}/Sites", :nfs => nfs_status #Put some cached files on the host to save downloading on rebuild config.vm.share_folder "chef-cache", "/var/chef/cache", "#{ENV['HOME']}/Sites/tmp/chef/cache" config.vm.share_folder "chef-file-store", "/srv/chef/file_store", "#{ENV['HOME']}/Sites/tmp/chef/file-store"   #Common config shared between multiple VMs def shared_chef_config(chef) chef.chef_server_url = "https://api.opscode.com/organizations/inviqa" chef.validation_key_path = "~/Sites/inviqa/tools/chef/.chef/inviqa.pem" chef.validation_client_name = "inviqa"   # Set the environment for the chef server chef.environment = "development"   # Put the client.rb in /etc/chef so chef-client can be run w/o specifying chef.provisioning_path = "/etc/chef"   # logging # chef.log_level = :debug end   def set_node_name(chef, node_name) chef.node_name = "#{ENV['CLIENT_KEY_NAME']}-#{node_name}" end   #Web VM config.vm.define :web do |web_config| web_config.vm.network :hostonly, "192.168.10.50" web_config.vm.provision :chef_client do |chef| shared_chef_config chef set_node_name(chef, "web") chef.add_role("web") end end   #API VM config.vm.define :api do |api_config| api_config.vm.network :hostonly, "192.168.10.51" api_config.vm.provision :chef_client do |chef| shared_chef_config chef set_node_name(chef, "api") chef.add_role("api") end end end

knife_ec2 - Not the Sharpest Tool in the Box

I've been doing some work with chef and EC2 and have found a few idiosyncrasies that can make it challenging to use. It works really well for what it does, firing up EC2 instances and bootstrapping chef onto them. Where I ran into problems was trying to script EC2 instances to start using knife ec2 server create. I'd gotten fed up of typing a command line as long as my arm and, as is sensible, put the defaults into my knife.rb config file. Once I was happy with how it was working, I sprinkled a few of these commands judiciously through a script and ran it. These calls to knife were config driven, so they had the full command line with different options to start different AMIs, different AMIs in different regions along with various other things.

To my surprise, all these options were blissfully ignored in favour of my config file.

This behaviour was so unexpected that it took me a little while to spot it. I was busily debugging my script thinking I'd made a typo somewhere, or was loading the wrong config. Hopefully, forewarned will be forearmed and you won't fall into this trap, unlike the many others who have previously when I raised it on the chef mailing list.

If you want to keep an eye on it, there is a bug raised against it on the Chef bug tracker. Time permitting, I hope to brush up on my Ruby skills and pitch in to help on this, rather than just moaning about what is otherwise a great knife plugin, but I'm happy for someone else to get there first.

Does Your Source Control Match Your Deployed Recipes?

When working with chef, you quickly find out the pitfalls of having changes made through the opscode GUI and changes being pushed to your organisation via knife. With good discipline you can push all changes from files via knife and commit the changes back to source control. However, we all know other developers who are not as rigorous as us. If you don't this may be you, dear reader, and it's time to mend your ways.

Whether you're a lone developer trying to implement good practices, or working on a larger project where you're trying to enforce encourage them, health_inspector is a useful tool to do this.

It allows a quick comparison between your local repository and your chef server to check that your files are in sync. It's a relatively young project, so may be a little rough round the edges, but the developer is active and responded quickly when I identified a few bugs, fixing them quickly and providing a new release.

When discussing this with a colleague, he mentioned a useful knife plugin for bringing out of sync chef repos back into sync, knife-align. The guys behind that have also developed knife-consistency which looks to do a similar job to health_inspector in a knife plugin. I've not had a chance to look at it either of these yet, but being aware of all the tools at hand puts you in a better position to solve the problem, should it arise.

Over to You!

Are you doing something interesting with Vagrant and/or Chef? While we've been using these tools for a while now, we're still refining our approach and sharing our findings between projects. By sharing them with you, hopefully you'll figure out new things to do and share them with us. If you've got any interesting techniques or improvements on those listed above, please comment as we're always looking for new and interesting uses for technology.

If this sounds like the sort of thing you're interested in doing, Inviqa are hiring across the UK, so drop us a line at careers@inviqa.com and come and contribute.

                          
]]> http://techportal.inviqa.com/2012/06/26/tips-for-vagrant-and-chef/feed/ 2 http://techportal.inviqa.com/2012/05/11/dpc-ticket-winner-announced/?utm_source=rss&utm_medium=rss&utm_campaign=dpc-ticket-winner-announced http://techportal.inviqa.com/2012/05/11/dpc-ticket-winner-announced/#comments Fri, 11 May 2012 10:28:38 +0000 Lorna Mitchell http://techportal.ibuildings.com/?p=3786 We are delighted to announce that our competition winner is Christian Schorn! Christian wins a pass which gives him access to the conference sessions for both DPC (Dutch PHP Conference) and DMC (Dutch Mobile Conference).

Thanks to everyone who left awesome comments on the post, we loved hearing your stories! If you didn't win, we still have some places available, so buy your tickets and we'll see you there!

                          
]]> http://techportal.inviqa.com/2012/05/11/dpc-ticket-winner-announced/feed/ 2 http://techportal.inviqa.com/2012/05/09/conference-report-whisky-web/?utm_source=rss&utm_medium=rss&utm_campaign=conference-report-whisky-web http://techportal.inviqa.com/2012/05/09/conference-report-whisky-web/#comments Wed, 09 May 2012 10:16:40 +0000 Marco De Bortoli http://techportal.ibuildings.com/?p=3775 Whisky Web is a brand new, fresh conference and the 2012 edition was the inaugural event, with hopefully many more to follow. This event has some familiar names from the PHP community behind it, Juozas "Joe" Kaziukėnas and his helpers Michael Maclean, Max Manders, Dale Harvey and Paul Dragoonis.

The opening keynote from Josh Holmes was about learning how to fail, because everyone has to fail, not once but many times, because it's only by failing that we learn. So please never be afraid to make mistakes and never be afraid to ask for help. That's a great way to grow up personally and professionally in this industry.

In contrast, David Zuelke's closing keynote was deeply technical. He talked about RESTful web services and how to implement them. What I liked was the fact that in contrast to the current trend, where it seems JSON is the only answer to data transfer across web services and systems, David pushed heavily towards XML and the fact that XML is more descriptive and comprehensive than a JSON data structure. He made the important point that while XML can easily represent a JSON set of data, the opposite is not necessarily true.


1st talk - Estimation, or "How To Dig Your Own Grave"

Rowan Merewood is not only one of my colleagues but also one of my favourite speakers. He pointed out the most common attitudes towards estimation within companies in the past, and how this affects the estimation process and the likely consequences. He then gave some solutions, some different perspectives, explained how agile can help and why.

During the questions section this talk higlighted how managers, as well as developers, are interested in attending these events. I think this is a good indicator that the two worlds may yet converge to a common point of understanding.

2nd Talk - Essential Node.js For Web Developers

Mike Amundsen has all my appreciation for delivering a really comprehensive talk about Node.js. He demonstrated practical usage of Node.js in the web ecosystem, using it to render web pages and forms. He also gave insight on how to deal with getting and posting requests and even how to handle file uploads and downloads and content streaming. This talk drew on the personal experience of one person and transformed it to motivate people to have a deeper look at Node.js and its capabilities. Thumbs up for Mike and for Node.js.

3rd Talk - The Emperor's New Clothes

Kevinjohn Gallagher's talk was a very good one, but at the same time probably the one I liked the least, simply because his topic was huge and we ended up overlooking and passing very quickly through some slides that in my opinion deserved more time and attention. In an era where everyone talks about new methodology, innovation, HTML5, responsive design and everything other thing that pops into your head when you think about what are the current technical hypes, Kevinjohn explained how this rush to a brave new world is not always helpful. He presented evidence showing that often, this tendency to upgrade and change technologies is quite unnecessary. It does not benefit the end user, it's a huge investment for a company, and sometimes it is done just for the "fun" of it.

4th Talk - Lessons Learned From Testing Legacy Code

John Mertic covered another very interesting topic that has touched me personally in the last few years of my career. Generally, when you go to conferences or you work abroad or you do consultancy you meet either of two scenarios: The first scenario is that of the cool people talking about testing and best practices. The second, more common, scenario is where unit test coverage is thin, where the software is badly designed, and the basic principles of object orientation are just a dream! John refreshed our knowledge and skills on what the available tools are; how, when and why to use them in order to achieve good quality results. I'm sure this session was enlightening for many people because so much of what he covered seems unknown to so many developers.

5th Talk - Is Your App Ready For The (Hybrid) Cloud?

Thijs Feryn is a proactive and engaging speaker, on the stage he tries to take the audience aboard with him in his adventure through the slides. He works for a hosting provider so getting in touch with architectural solutions and distributed systems seems to be a daily task. During his talk he described some of the cloud solutions that the current market has to offer. He talked about Amazon's Web Services, Microsoft's Azure platform, Orchestra's infrastructure and many more - giving a detailed overview of the pros and cons of every solution, depending on the requirements of your project. The Q&A phase of this talk was full of interesting thoughts and possible solutions to investigate.

Wrap up

Attending a conference is absolutely key for any developer's professional career, knowledge and personal development, the return on your investment is very high and I cannot recommend strongly enough to people to attend a conference! This particular conference had a nice format: one day of talks and one hackathon day.

At the social event on Friday, the conference was true to its name - we had a whisky tasting night. This was the cherry on the cake and it allowed me to taste for the first time in my life a type of gin which I liked!

                          
]]> http://techportal.inviqa.com/2012/05/09/conference-report-whisky-web/feed/ 0 http://techportal.inviqa.com/2012/04/27/dpc-radio-keynote-the-art-of-the-user-experience-making-beautiful-delightful-fun-things/?utm_source=rss&utm_medium=rss&utm_campaign=dpc-radio-keynote-the-art-of-the-user-experience-making-beautiful-delightful-fun-things http://techportal.inviqa.com/2012/04/27/dpc-radio-keynote-the-art-of-the-user-experience-making-beautiful-delightful-fun-things/#comments Fri, 27 Apr 2012 08:22:40 +0000 dpcradio http://techportal.ibuildings.com/?p=3398 Aral Balkan

We are the makers of the new everyday things. We design and develop the virtual pens, telephones, newspapers, calendars, and door-handles that people interact with every single day. We are the virtual architects and the products that we design and develop have the power to determine whether people have a good day or a bad day.

In this session, Aral Balkan will outline the important role that user experience design plays in the making of virtual products and inspire you to see that it is your job – regardless of whether you make web sites, mobile apps, intranet systems, or ticket machines – to make this new world that we are crafting together not only usable and accessible but beautiful, fun, inspiring, pleasurable, delightful, and – dare I say – magical.

                          
]]> http://techportal.inviqa.com/2012/04/27/dpc-radio-keynote-the-art-of-the-user-experience-making-beautiful-delightful-fun-things/feed/ 2 Aral Balkan,dpc11,inspiring,keynote,user experience Aral Balkan We are the makers of the new everyday things. We design and develop the virtual pens, telephones, newspapers, calendars, and door-handles that people interact with every single day. We are the virtual architects and the products that we des... Aral Balkan We are the makers of the new everyday things. We design and develop the virtual pens, telephones, newspapers, calendars, and door-handles that people interact with every single day. We are the virtual architects and the products that we design and develop have the power to determine whether people have a good day or a bad day. In this session, Aral Balkan will outline the important role that user experience design plays in the making of virtual products and inspire you to see that it is your job – regardless of whether you make web sites, mobile apps, intranet systems, or ticket machines – to make this new world that we are crafting together not only usable and accessible but beautiful, fun, inspiring, pleasurable, delightful, and – dare I say – magical. techPortal no 58:08