Thanks to everyone who left awesome comments on the post, we loved hearing your stories! If you didn't win, we still have some places available, so buy your tickets and we'll see you there!

                          

The opening keynote from Josh Holmes was about learning how to fail, because everyone has to fail, not once but many times, because it's only by failing that we learn. So please never be afraid to make mistakes and never be afraid to ask for help. That's a great way to grow up personally and professionally in this industry.

In contrast, David Zuelke's closing keynote was deeply technical. He talked about RESTful web services and how to implement them. What I liked was the fact that in contrast to the current trend, where it seems JSON is the only answer to data transfer across web services and systems, David pushed heavily towards XML and the fact that XML is more descriptive and comprehensive than a JSON data structure. He made the important point that while XML can easily represent a JSON set of data, the opposite is not necessarily true.

1st talk - Estimation, or "How To Dig Your Own Grave"

Rowan Merewood is not only one of my colleagues but also one of my favourite speakers. He pointed out the most common attitudes towards estimation within companies in the past, and how this affects the estimation process and the likely consequences. He then gave some solutions, some different perspectives, explained how agile can help and why.

During the questions section this talk higlighted how managers, as well as developers, are interested in attending these events. I think this is a good indicator that the two worlds may yet converge to a common point of understanding.

2nd Talk - Essential Node.js For Web Developers

Mike Amundsen has all my appreciation for delivering a really comprehensive talk about Node.js. He demonstrated practical usage of Node.js in the web ecosystem, using it to render web pages and forms. He also gave insight on how to deal with getting and posting requests and even how to handle file uploads and downloads and content streaming. This talk drew on the personal experience of one person and transformed it to motivate people to have a deeper look at Node.js and its capabilities. Thumbs up for Mike and for Node.js.

3rd Talk - The Emperor's New Clothes

Kevinjohn Gallagher's talk was a very good one, but at the same time probably the one I liked the least, simply because his topic was huge and we ended up overlooking and passing very quickly through some slides that in my opinion deserved more time and attention. In an era where everyone talks about new methodology, innovation, HTML5, responsive design and everything other thing that pops into your head when you think about what are the current technical hypes, Kevinjohn explained how this rush to a brave new world is not always helpful. He presented evidence showing that often, this tendency to upgrade and change technologies is quite unnecessary. It does not benefit the end user, it's a huge investment for a company, and sometimes it is done just for the "fun" of it.

4th Talk - Lessons Learned From Testing Legacy Code

John Mertic covered another very interesting topic that has touched me personally in the last few years of my career. Generally, when you go to conferences or you work abroad or you do consultancy you meet either of two scenarios: The first scenario is that of the cool people talking about testing and best practices. The second, more common, scenario is where unit test coverage is thin, where the software is badly designed, and the basic principles of object orientation are just a dream! John refreshed our knowledge and skills on what the available tools are; how, when and why to use them in order to achieve good quality results. I'm sure this session was enlightening for many people because so much of what he covered seems unknown to so many developers.

5th Talk - Is Your App Ready For The (Hybrid) Cloud?

Thijs Feryn is a proactive and engaging speaker, on the stage he tries to take the audience aboard with him in his adventure through the slides. He works for a hosting provider so getting in touch with architectural solutions and distributed systems seems to be a daily task. During his talk he described some of the cloud solutions that the current market has to offer. He talked about Amazon's Web Services, Microsoft's Azure platform, Orchestra's infrastructure and many more - giving a detailed overview of the pros and cons of every solution, depending on the requirements of your project. The Q&A phase of this talk was full of interesting thoughts and possible solutions to investigate.

Wrap up

Attending a conference is absolutely key for any developer's professional career, knowledge and personal development, the return on your investment is very high and I cannot recommend strongly enough to people to attend a conference! This particular conference had a nice format: one day of talks and one hackathon day.

At the social event on Friday, the conference was true to its name - we had a whisky tasting night. This was the cherry on the cake and it allowed me to taste for the first time in my life a type of gin which I liked!

                          

We are the makers of the new everyday things. We design and develop the virtual pens, telephones, newspapers, calendars, and door-handles that people interact with every single day. We are the virtual architects and the products that we design and develop have the power to determine whether people have a good day or a bad day.

In this session, Aral Balkan will outline the important role that user experience design plays in the making of virtual products and inspire you to see that it is your job – regardless of whether you make web sites, mobile apps, intranet systems, or ticket machines – to make this new world that we are crafting together not only usable and accessible but beautiful, fun, inspiring, pleasurable, delightful, and – dare I say – magical.

                          

This year, we're running a sister conference alongside DPC - the very first Dutch Mobile Conference is being held! Your DPC ticket also gives you access to the mobile conference tracks, so whether you're into web, mobile, or a little bit of both - you're absolutely in the right place :)

We have some excellent speakers lined up for both conferences; check out the mobile conference schedule and the PHP conference schedule to see what kinds of sessions are available.

How to Win the Ticket

We're keeping it nice and simple! To win, simply leave us a comment on this post (including your email address so we can get in touch with you if you win!) telling us why you should win the ticket, and we'll randomly draw a winner when we close the competition on May 9th. Want to join us in Amsterdam? Then what are you waiting for?

(sadly we haven't got enough tickets to give away to everyone, so if you don't win then you can still buy tickets and we'll see you there!)

                          

Contest Key Rules: A Recap

• You should follow the Symfony code standards as described here.
• Your script will be run on PHP 5.3.5 installation. Should you need any PHP extension you must provide a patch to the php.ini with the changes made to add the extensions you've used.
• In the spirit of Sismo itself, we'd prefer self-contained solutions. Use Symfony or PHP extensions if you need to, and other frameworks if you must!
• The name of the PHP class doesn't matter, but it must extend Sismo\Notifier and implement the notify() method.
• You can look at existing Sismo notifiers to see implementation examples, but you cannot implement the same notifier in a different way, e. g. you cannot implement a Growl notifier which is already in master. You can find more instructions on how to extend the notifier on Sismo documentation. It can be found in sismo.sensiolabs.org or github.com/fabpot/Sismo under “Adding a notifier”.

Competition Entries

We created 3 categories; creative, inspiring and useful notifiers. None of the submissions selected a category, so we chose the category we thought best described the submission. We received a total of 23 subscriptions, but only 14 of those actually submitted their code or a link to an existing repository.

Winners

The competition was designed to inspire developers to contribute to open source projects. We specifically chose a tool that requires existing unit tests for you to use it – a continuous testing tool - to encourage contestants to write tests. Above all, I hope you have enjoyed taking part in this contest. Now, let's look at the winners

Useful

Considering Sismo is meant to run locally we chose a Wallpaper Notifier as the most useful. It shows the build result as a small bar added to the desktop wallpaper. This is a very customisable solution; you have control over colours, the size of the final wallpaper considering the screen resolution, and a bunch of other options. The added documentation was also very good.

Category Winner Useful: Wallpaper Notifier by Javier Eguiluz

Inspiring

I have noticed that it has become common in some IRC channels to have a bot notifying the chat participants of the commits as they happen. One of the most popular of these bots is Eggdrop. So we picked an Eggdrop Notifier for the inspiring category. This would be particularly inspiring in open source projects that are just starting to build a community. The participants could see what the selected developers were working on locally.

Category Winner Inspiring: Eggdrop Notifier by Andrew Graham

Creative

We had a few submissions of notifiers that would speak the test results, but this one made me laugh. I guessed immediately what it was because of its name, but still it was very funny when it ran with a failed build: "Houston, we have a problem here!", it said. Nice, funny, creative and original.

Category Winner Creative: Houston Notifier by Andreas Hucks

Prizes

Amongst all the winners, we raffled three prizes: 2 tickets to the PHPNW Conference and an Apple iPad. The prizes go as follows:

PHPNW tickets: Andrew Graham and Andreas Hucks
iPad: Javier Eguiluz

Congratulations to all the winners!

Conclusions

For us as organisers, this has been another fun challenge and we look forward to the next (watch this space!). We offer our warm congratulations to all our participants for such a high standard of submissions, and especially to our winners who fought off some stiff competition to emerge victorious. Thanks to all of you and we hope you had as much fun as we did!

                          

Test-driven development is generally regarded as a good move: it should result in simple decoupled design, your tests tend to cover behaviour not methods, and far fewer bugs. However, just getting unit tests in on a real, commercial project is hard - switching to TDD is even harder. Defining concrete answers to a problem is hard and can be difficult to integrate into Often you can start a project with good intentions and coverage, then the deadline looms and the tests go out then the hacks come in. So, instead of beating ourselves up about not being perfect let's look at an interactive approach to adopting TDD principles. We'll look at tactics for selling TDD to your client, boss and colleagues. This talk will also cover methods for making TDD easier for you by showing you what tools you can use to integrate it into your development environment. In the project itself, we'll examine how we can make small but permanent steps towards full TDD, without losing that progress when deadlines hit. We'll also cover a few methods for learning on your own time and how the whole process can actually be made quite enjoyable.

You can find Rowan's slides on slideshare.

                          

Unfortunately, as with everything, this is not quite the case. Agile when utilised properly can have a positive effect on productivity, it can go some way to helping make deadlines easier to hit. But when used badly, it can have completely the opposite effect.

There are a number of problems, or 'smells', we have identified in the Agile process that could be having a negative impact on all your good intentions. Here we outline some of the smells you may find in various elements of the agile process and give you some tips on how to spot and eradicate them.

The Agile Process

This document anticipates a certain knowledge and experience of Agile, and so will not go into any depth explaining the processes and ceremonies of Agile. If you are looking for more of an introduction to Agile excellent guides can be found here and here

Sprint Planning Smells

The Sprint planning session, or sessions depending on how you approach it, is the foundation upon which you build the entire sprint. Any problems at this point in the process could be setting you up for failure before you even get started. Here are a few problems we have seen that can arise during these meetings.

Hit By A Bus

"I know exactly how to do that, give it to me and it's a 1 pointer"

There are always going to be developers who have more knowledge of certain areas of the application. The clichéd hit by a bus scenario, however, is a genuine risk; what if, once this 1 point story is added to the sprint, the "expert" developer is unavailable to do it? All of a sudden this story becomes a lot more complex due to having to find your way round code you don't know so well.

Tip:

• Estimate every story as if you will be doing it

Swayed By Giants

"They said it's a 1 point story, so I'm going to go with 1 point as well"

Every team is going to have a mixture of personalities and abilities, unfortunately this leads to a situation whereby some developers are reluctant to make their own estimates and instead allow their decision to be influenced by the stronger developers. The knock on effect is that the estimates are really only being done by one or two developers as the rest just agree. However the work is spread across the team, so these underestimated stories find themselves in the hands of developers who, while not estimating the story highly, believe it to be harder than their estimate.

Tips:

• As above, estimate every story as if you will be doing it
• If using planning poker cards, ensure everyone places the card face down first, ensuring that the initial estimate can not be influenced by others

Fuzzy Definition of Done

"That's only 2 lines of code, it's a 1 point story"

It may well be the case that the development of the story is trivial. But those 2 lines may be in such a core part of the architecture that the QA effort required to validate it will take days and days. With a formally defined Definition of Done, this can outline the steps required before a story can be marked complete. One of the key aspects of that should be, I am confident this code could go to production now.

Tips:

• Create a formal Definition of Done for the project
• When estimating estimate against this definition of done, not merely the development effort required

Anonymous Author's task

"As a user I need better reporting".

We have all seen stories that look a little like this. The problem is that this makes perfect sense to the author, but little sense to anyone else. If the author is unavailable during the planning poker session then you are left with two, fairly undesirable, options. You can either reject the story outright, citing a lack of detail. This is probably the better option here in order to protect the sprint, however it does go against the philosophy that we are all working together. The other option is to try to guess what you think the author wanted and estimate that. Which could threaten the sprint once it gets accepted if you have misinterpreted the story, particularly if underestimated

Tips:

• Ensure that stories go through a validation process before they reach the Planning Poker session. This can help avoid vague or incomplete stories reaching the session.
• If possible try to ensure that the author of the story is present in the session to answer any further questions about it

Premature Architecture

"We could implement a simple form here, but we'll have to be wary of SQL Injection"

It really is no bad thing to be wary of SQL Injection, in fact it is to be positively encouraged. However this is not the forum to be having these kind of discussions. At this moment all we need to understand is how complex this story is, and thus how much relative effort will be required to complete it.

On occasion the manner in which the story will be tackled will need to be defined early, for example the problem could be solved by hardcoding a value in a page, or by creating an editable admin form. Clearly here the approach taken influences the estimate, but ideally this knowledge will be known before the planning poker session

Tips:

• Attempt to vet potentially contentious stories before they reach the planning poker session
• Consider the complexity of the solution, but not the finer details of it

Ramble on

"I think we should consider… then of course we need … but what about the case … then users in Tibet might find …"

Often in your planning poker session there will be a number of stories requiring estimation. Occasionally you will find the sort of scenario described above where the requirements are not quite specific enough and so the discussion goes back and forward with lots of questions and answers which either stretch this session out for longer than it needs to be, or meaning that some stories don't get estimated as the time runs out

Tips:

• Ensure stories are clearly defined to avoid this kind of waffle
• Timebox each story, if an estimate can't be reached in a reasonable amount of time then there are issues with the story

Standup Smells

The standup can be a somewhat misunderstood part of the sprint process. A common misconception is that each of the team members are reporting back on what they have been doing to their "superiors". In fact it is quite the opposite, and should be viewed as such. The team is working towards a common goal, and the standup is the ideal opportunity to ensure that as a team you still are on course to reach that target. You are not reporting upwards, but rather sideways, ensuring that your team members are up to date with how the sprint is progressing from your point of view, while finding out how the sprint is progressing rom their point of view. Problems identified early, in this forum can be handled with a team effort ensuring any blockers get prioritised for the team's common good. This leads us to our first standup smell.

Scrum Master Centred

"Yesterday I worked on X, today I will work on Y"

It's not clear from the above quote (I couldn't think of one that worked!), but the inference here is that the team member is ‘reporting' on their day's work to the Scrummaster, or Product Owner. The important thing to remember in the daily standup is that it is a forum for information sharing amongst the team. Not a mechanism for people keeping track on your progress.

Tip:

• Attempt to look at every single team member while you are giving your update

Turn Up, Tune In, Drop Out

"what, er, me? Oh ok. Er, yesterday I …"

Possibly as a consequence of the previous smell, namely Scrum Master Centered, because people treat the meeting as reporting to their ‘superior', you quite often see the consequence is that people are essentially just waiting their turn to speak and not paying attention to the remainder of the team.

Tips:

• Mix up the order, perhaps using a ball, or other similar object, that the team pass to each other in a completely random order. This way anyone could be called upon to speak at any point.
• Remember this is much about you learning what your teammates are doing, as them finding out what you have been doing.

Ramble On

"So I started looking at … then I thought I might try … it didn't do what I thought, but that was interesting … then it turned out that …"

This is not the forum to elaborate extensively on a particular thing you were working on, or a particular problem you faced. The standup is supposed to be a very short meeting, hence standing up for it. The 3 main questions each person should answer in a stand up are traditionally, "What I did yesterday", "What I'll do today" and "Is anything blocking me". You should be able to get through those in under 30 seconds

Tips:

• Timebox each speaker
• Try to make your points concisely
• If necessary, have a further conversation with relevant members of the team outside of the standup environment

Hidden Impediment

"Today I'll work on story Y, should take a few hours"

"Yesterday I worked on Story Y, I'll keep doing that today"

"Yesterday I was still on story Y, today I'll keep at it"

You'll notice here that on the first day, the developer mentioned that he anticipated finishing the story within a few hours, yet 3 days later is still plugging away at it. This suggests that something unanticipated has come up and is getting in the way of the developer who, in this instance, instead of asking for assistance is doggedly working away at the task in hand. Clearly now this developer is behind schedule (having spent 2 unanticipated days working on this problem) and the knock on effect is that the sprint itself could be in danger.

Tip:

• Alter the language of the daily standup questions slightly, from what I will work on, to what I will achieve. This way, in the scenario above, the developer would have said on the first day that he would achieve story Y and when he hadn't on the second day it would be more obvious that there may be a problem with this story.

Off Topic

"Did you see the football game / reality show result / interesting phenomenon last night?"

It may well be that the football game / reality show / phenomenon was the most fascinating thing to happen to the planet this millennium. But there will be all the time in the world after the standup to discuss it. The standup should be focused entirely on the sprint progress.

Tip:

• Stay on topic

Memory Problems

"I can't remember what I was working on yesterday, but today I'll pick up some task"

This is particularly common on the first standup after the weekend! These updates are of no use to anybody as they completely fail to address the overriding concern of the standup which is, "Are we still on track to deliver this sprint on time?"

Tip:

• Take 2 minutes before the stand up to reacquaint yourself with what you were working on, and what is likely to be your focus today

Problem Solving

"What was in the error logs with that problem you have? "

It may well be the case that you are struggling with a particular problem, and you mention that as part of the standup. The danger here is that the Standup gets taken over by an improptu problem solving exercise. This may not sound like a bad thing, after all we have the team together why not utilise that fact to try to get to the heart of this problem? The simple answer is that this is not the function of the standup, and it's rarely the case that the entire team needs to be involved in this subsequent discussion. Identifying the problem, and identifying who can have input into the solution is the domain of the standup, but this actual discussion should then take place separately, allowing the standup to continue

Tip:

• Identify who can help solve the problem, then have a discussion after the standup

Talking Chicken *

"When will this feature be ready to test, it really is critical to our business"

Anybody related to the project should be welcomed, indeed encouraged, to attend the standup. However, as we have mentioned before, we need to remember the purpose of the standup and that is for the team to catch up with each other to get a sense of how the sprint is going. Just as this is not the team members reporting to a scrum master, it is also not the team reporting to an external stakeholder. An external stakeholder, such as a Product Owner, may be treated as a team member and provide an update of their own to the team, however they should not be given the licence to ask direct questions, such as "When will feature Z be ready", or more dangerously, "What if it worked that way instead…". The dangers of this range from the standup losing its purpose, through to scope creep, which we shall touch on later in the article.

* See the analogy of the chicken and the pig for the reasoning behind the title here

Sprint Smells

Now that we have got planning out of the way, and are happily working through the backlog as a team there are still a number of smells that can arise which threaten the delivery of the sprint. We analyse these in detail here.

Gold Plating

"I know how I can write this using [INSERT CODING TECHNIQUE DU JOUR]"

As developers we are always seeking a technical challenge, and ways to demonstrate our ability. However if left unchecked, these desires can lead to us unwittingly putting the sprint under threat by adding unnecessary flourishes to otherwise trivial pieces of work. As a deliberately ridiculous example, think of a story which has been raised to provide a couple of text changes to page on a website. Having been bogged down with stories such as this previously, you take it upon yourself to implement a CMS system letting the product owner make such changes in future. All in the best of intention aimed at helping save time ultimately. The only problem being that this initial story which was a trivial 1 pointer has now evolved into a completely new story, taking much more time and effort, and now meaning that the current sprint is under threat as the remainder of your tasks are unlikely to be completed in time.

Tips:

• Solve the current problem
• Don't try to predict and solve every eventuality

Pulling the thread

"I don't like the way this is written, I'll just refactor it a little…"

Similarly to the above, it may be the case that you are working in a certain area of the codebase and you notice a piece of code that is far from optimal, and you can instantly see how to improve the code and perhaps improve the performance or security of this part of the application. It could even be the policy of the team that if you are working on a piece of code with insufficient code coverage or quality then it is your duty to raise that, even if it doesn't directly correlate to your story. However the underlying danger here is that you start refactoring a piece of code and three days days later you are still trying to untangle code that is now so far removed from your initial story as to be unrecognisable.

Tips:

• Try to consider the current story complexity when considering refactoring. If the refactor effort will add a large overhead, then this is not the time for that
• Write a story detailing the refactoring required and try to ensure that it gets prioritised as appropriate

Scope Creep

"Well that is really good, but wouldn't it be great if it did this…".

One of the main advantages of Agile is being able to react to change, and to be open to this kind of request. However the sprint is a contract between the team and the product owner, and any change to this contract needs to go through a formal change process. This can seem overkill when the request is relatively trivial, but it is so often the case that when you do something once, it's difficult to justify not doing it again. This is particularly an issue if the Acceptance Criteria are set, and this new change would go against these. Having a formal change review process would have the acceptance criteria altered as part of the process, meaning there is no cause for confusion between the developer and the quality assurance team. Allowing scope creep also may add complexity to the task, meaning there is a threat that the sprint will fail to be delivered due to this additional overhead.

Tips:

• Ensure that once the sprint has started that there is a formal process to go through should anything need changed
• Accept that these changes can happen, but ensure the project owner is aware that something already prioritised in this sprint may have to give to accommodate this additional change

Retrospective Smells

The retrospective is arguably the most important ceremony within the Agile process. It allows the team to dissect the previous sprint and analyse how they felt it went. Those aspects of the sprint that went well should be highlighted, allowing the team to self congratulate on a job well done, and actions should be made to attempt to continue doing the things that led to those successes. However, at least as important is the highlighting of those aspects of the sprint that did not go so well. Again, actions should be raised to attempt to eradicate any obvious problems that are contributing to these failures.

How to run a successful retrospective is beyond the scope of this document, however some good guides can be found here and here.

Utilised properly, the retrospective is a potent weapon in getting your team gelling and working well together. However falling prey to any of the smells listed below can result in the retrospective being little more than a waste of everybody's time

Practising Insanity

"We'll not bother with the retrospective this week"

We have all heard the phrase, widely attributed to Albert Einstein, that the definition of insanity is doing the same thing over and over and expecting different results.

Very rarely do you achieve the perfect sprint. There is always something waiting to trip you up. Unless you take the time to analyse how the problem came about, you will frequently find the same problem coming back to bite you again and again.

Quite often the retrospective is the first thing to get sacrificed when deadlines start looming. Unfortunately this removes the team's ability to attempt to get to the root of any problems that may occur during the sprint. Without taking the time to acknowledge the problems, they will get ignored and there will always be the threat of these problems reoccurring in later sprints.

Tip:

• Stick to your guns. It's going to be a difficult sell, but it is better to miss delivery of this sprint and ensure that the lessons learned get addressed, rather than sacrificing the retrospective for an extra few man hours of development only to get tripped up by exactly the same problems again

Lights..Camera..CUT

"Ok, good retrospective folks, shall we go to the pub?"

The second common problem with retrospectives is that they happen, they are productive, a lot of good content and self reflection happens within the context of the meeting, and then nothing happens. If you are going to the trouble of having a retrospective, then ensure that the outcomes of the retrospective get carried forward into not only the next sprint, but all future sprints. There are a number of techniques you can employ to ensure that you are making the most of retrospectives.

Tips:

• At the start of the retrospective, analyse the action points you came up with in the previous retrospective. Ensure that you have indeed been following up on these action points, to the point that they are becoming second nature. If you feel that you are still not naturally performing a particular action point ensure it gets carried across into this sprint's action points
• Assign one or more action points to each member of the team, and make it their responsibility to ensure these action points are actioned over the course of the sprint

Conclusion

Agile itself is not a silver bullet that will instantly solve all your development woes overnight. It's not necessarily even the best approach for your particular team. It is, however, a great methodology for fast reactive web design when utilised correctly.

As we have just seen, just blindly following the ceremonies and calling it Agile is not in itself going to guarantee any improvement in output, and could even have the opposite effect. With any luck you will not recognise too many of the items mentioned above in your own processes and you already have a high functioning team. Otherwise, hopefully this article will help you spot some problems in your process and assist you getting the most out of your team.

                          

Even bad code can function. But if code isn't clean, it can bring a development organization to its knees. Every year, countless hours and significant resources are lost because of poorly written code. But it doesn't have to be that way. In this session you will learn how you can offset your technical debt with clean code that is readable and testable as well as reusable.

                          

Did you know there are alternatives? A NoSQL database is one of them, and it has its use cases, but that will need some consideration, and almost certainly some rewriting of your application, to take advantage of. Instead, let me talk to you about some databases that speak your applications language, namely MySQL's client server protocol.

A Quick Recap

MySQL AB was founded by Michael Widenius (Monty), David Axmark and Allan Larsson in 1995. After the 3.x and 4.x release series, MySQL 5.0 arrived on our servers in 2005, bringing in big changes with new features and improved performance over the 4.x releases. In 2008 Sun Microsystems acquired MySQL AB. After being critical of the release process for 5.1, Monty left Sun in 2009 to form Monty Program AB. Also in 2009, Oracle made a bid to acquire Sun, which reached final approval in 2010. Since then, Oracle have released MySQL 5.5 bringing scalability and monitoring improvements.

These changes of ownership and personnel changes, amid concerns over the direction of MySQL development, have sparked off a thriving community of databases based on the open source MySQL code, developed by people who have strong ideas of the direction they' would like to see MySQL take.

The Alternatives

Percona Server with XtraDB

Percona was founded by Peter Zaitsev, one of the worlds foremost experts in high performance MySQL, having run the "High Performance MySQL" group within MySQL AB's support group from 2002 to 2006. Percona Server is a patched build of MySQL with Percona's XtraDB storage engine, which itself is based on MySQL's InnoDB storage engine. The patches applied have been developed to solve real world problems encountered by the Percona team. These patches are readily available and quite often are integrated back into MySQL releases. The XtraDB storage engine aims to improve the performance and scalability above that of the InnoDB storage engine by taking advantage of hardware features that weren't available or were relatively uncommon when InnoDB was first designed, such as multiple processor cores.

MariaDB

Monty formed Monty Program AB to, amongst other things, continue developing MySQL in the direction he thought it should go. To achieve this, he forked MySQL and named the fork MariaDB. Key aims of the MariaDB project are to create a more open development model to encourage community participation and to provide a complete drop-in replacement for MySQL. Features from the MySQL code base are regularly merged to bring in fixes and to help maintain compatibility, although there has been some criticism of how quickly this happens. On top of this MariaDB looks to improve performance and add new and better features.

Drizzle

In 2008 Brian Aker, who was previously MySQL's Director of Architecture, forked MySQL to create the Drizzle project. The Drizzle project follows a similar process to the development process of the Linux Kernel, promoting control by the community over control by a company, both at the code level and at the developer level. It's designed as a pluggable micro-kernel promoting easy extension. While Drizzle was initially forked from the MySQL 6.0 code base, effort has been made to favour performance over compatibility with MySQL, particularly with reference to scalable performance for multiple cores and cloud deployments.

Which one should I use?

After reading this article, resist the temptation to go running off and install one of the alternatives on your production hardware! If after reading this article you do decide to investigate migrating, I would highly recommend benchmarking the performance, stability and compatibility of the alternatives against your application. In fact, it is less of a recommendation and more of a requirement. If you take anything away from this article, it should be that each of the teams have concentrated on different areas of MySQL. Which one works best for you depends on your application workload. What works best for sustained throughput of short running simple queries isn't necessarily going to work best for lots of long running complex queries. Whether they continue to diverge to fulfil different needs or whether one of the projects manages to successfully integrate the best features of all projects to create an über-database remains to be seen. If that happens, then you get to have it all, but for now to find the best database for you, you will need to benchmark, benchmark, benchmark!

Compatibility

Percona Server and MariaDB

Both Percona and MariaDB are binary compatible. You can grab a tarball of either, extract it stop MySQL, start Percona/MariaDB, load up your client and start running queries like you'd just restarted MySQL. This is a great starting point to test your application and see what improvements you get out of the box, it is also very good for benchmarking as it means you don't have to migrate your data to make comparisons.

Drizzle Compatibility

The Drizzle acorn has fallen furthest from the oak that is MySQL. While they share common ancestry, Drizzle is no longer binary compatible with MySQL and therefore more care needs to be taken over migration. Here's what most people will need to know:

• Drizzle supports InnoDB, but the InnoDB file formats are different. This means you have to transform your data, you can't just start Drizzle and point it at your MySQL files.
• MyISAM is supported only for explicitly created temporary tables, and is expected to be completely unavailable in a future release.
• MERGE, FEDERATED, ARCHIVE and CSV storage engines have all been removed.
• Triggers and views aren't supported natively, however appropriate hooks are provided to allow these to be added as plugins
• MySQL's stored procedures implementation has been deemed to be less than optimal and has been removed. There's no indication that this will be reimplemented, they are favouring improved batching of commands over the wire to reduce the time row locks are held, which is the main advantage of stored procedures in MySQL.This decision also fits with the Drizzle aim to reduce database complexity.
• Some functions, commands and keywords have been removed
• Some objects have been removed. A rule of thumb as to what has been removed is where there were previosuly TINY-, SMALL- and MEDIUM- types, there is now one optimised type. Aliases have also been removed, e.g. SET has been removed leaving ENUM. A comprehensive list of removed objects is available.

Most of these changes are likely to affect your database administration more than your application implementation, sticking to the idea that you should still be able to drop Drizzle in with little or no change to your application.

To simplify the migration of data from MySQL to Drizzle, Drizzle provide the drizzledump tool. While designed for Drizzle backup, it can connect to a MySQL server and produce a Drizzle-compatible export. It manages the above compatibility issues for you, e.g. conversion of data types in table definitions. If you've got both servers live, drizzledump can even import from MySQL straight into Drizzle without an intermediary file.

Features

Not only do the alternatives offer improved performance for some use cases, but they also offer additional features that are not present in MySQL. Deciding to take advantage of these features means you move further away from MySQL compatibility and portability, but if there is a killer feature you want to use in your application then adopt the alternative but be aware of this trade-off. While you're still deciding on the best database server for you, you need to track and isolate these changes to allow you to switch between the alternatives. I'd suggest creating a branch, or equivalent, in your version control system for application change. It's worth version controlling your database config as well, if you're not already (you can read more about this in Harrie's post about database version control). Even better you could use a DevOps tool like chef or puppet to create reproducible database deployments, but that's a subject for another article.

Percona Server with XtraDB

XtraDB

Percona Server comes with XtraDB, a drop-in replacement for the InnoDB storage engine. It offers compatibility with InnoDB, so everything you can do with InnoDB works with XtraDB. The differences are improved performance, which we'll cover shortly, diagnostics and reliability. If you've ever had problems due to corrupt InnoDB tables you'll be pleased to know that XtraDB gives you a corrupt table error rather than a crash.

HandlerSocket

HandlerSocket is a MySQL daemon plugin that gives you a NoSQL interface to MySQL storage engines. It is fast! Benchmarks have shown it supporting nearly twice the number of queries per second as memcached. That's right, not twice as fast as MySQL client access (it is nearly 7 times as fast actually) but nearly twice as fast as memcached.

Percona Server comes with the HandlerSocket plugin built and ready to run with just a couple of lines of config. The HandlerSocket plugin can be built for MySQL as well, but Percona Server ships with it ready to go and includes documentation on its configuration so, for that effort, it goes down as a feature for Percona Server.

Diagnostics and Monitoring

Percona's business is analysing MySQL installations with performance problems and providing solutions. This wouldn't be possible without being able to first identify the problem. While MySQL provides some indicators for status and performance, Percona found that they fell short so, during the course of their work, they've written many improvements and additions to the reporting of Percona Server status and performance. Percona Server now offers over 20 more tables in INFORMATION_SCHEMA and 60 more status and performance counters, along with breakdowns of performance per table, per index, per client and per thread, and this is still not a complete list of diagnostic capabilities added in Percona Server! A MySQL feature that you may be familiar with, the slow query log, has also had some additions, bringing microsecond precision to query run times, additional query statistics and the ability to enable and disable it at runtime. With this information more readily available, application and database optimisation becomes simpler, bottlenecks become easier to identify and problems are solved faster, all of which are as important to application development as squeezing better performance out of MySQL out of the box.

MariaDB

Additional storage engines

MariaDB offers additional storage engines which can provide optimisations for specific data models.

• Aria: Intended to be a fully transactional storage engine comparable with InnoDB and planned to be the default storage engine for MariaDB
• XtraDB: This is the same engine as discussed above and developed by Percona
• PBXT: A transactional storage engine originally developed by Primebase, now merged into the MariaDB development tree and maintained there
• FederatedX: A drop in replacement for the Federated storage engine
• OQGRAPH: A query engine designed to handle heirarchies and complex graphs. More information can be found at http://openquery.com/graph/doc.
• SphinxSE: While implemented as a storage engine, this plugin actually provides a connection to a sphinx search server.
• IBMDB2I:The MySQL IBMDB2I storage engine that was removed in MySQL 5.1.55

HandlerSocket and HANDLER statement

MariaDB also comes bundled with HandlerSocket as I mentioned above for Percona Server. It also takes direct storage access one step further by adding a HANDLER statement allowing for direct reads from storage engines via key lookup, allowing low level access that bypasses the query optimiser. This can offer up to 50% performance increase, but beware that it doesn't offer consistency. There are also a few cases where it might end badly; as always with great power comes great responsibility.

Virtual columns and dynamic columns

If you have used Oracle or MSSQL you may already be familiar with virtual columns. For those who haven't seen them before, they allow a derived column in your table. They operate in two modes, VIRTUAL and PERSISTENT. The behave the same but are implemented slightly differently. PERSISTENT virtual columns are calculated and written on row writes, whereas VIRTUAL virtual columns are calculated when the row is read. Which you use depends on whether you need to improve performance on read or write.

Dynamic columns allow you to store different columns for each row in a table. One use case for this is a products table holding multiple types of products, each with different attributes. Often this is handled by creating an EAV structure maintained by an application, however dynamic rows can provide a simpler solution (I know it looks like NoSQL, but it's not, promise ;) ). It is implemented by storing the dynamic columns in a blob and providing native functions for accessing this data. While the current implementation is stable and usable, there are some limitations and improvements are planned to address these limitations. The aim is to allow dynamic columns to be almost indistinguishable from normal columns.

Drizzle

Drizzle's focus on performance and simplification led to the Drizzle team designing a microkernel architecture, supporting plugins to provide features rather than implementing them in the core. This simplifies the process of adding features which will hopefully cultivate an ecosystem of Drizzle plugins and Drizzle plugin developers. It's clear that developing a thriving development community around Drizzle is the intention, as it's been made as simple as possible to develop and build plugins. As noted by Padraig O'Sullivan, this is 'freaking awesome'.

To date there are 75 official Drizzle plugins covering everything from authentication (Schema, LDAP, HTML, File, Allow all) to server protocols (MySQL, Drizzle) to storage engines (InnoDB, MyISAM, Memory). There is some criticism to this approach, as people are used to monolithic applications and expect to features to be available without needing to decide which plugins are enabled, with authentication being the usually-cited example. Personally I like the plugin approach, but it's not for everyone. As almost everything is a plugin, most plugins provide functionality that's present in the core of MySQL, so I won't go into those. Here are the best of the rest:

Authentication plugins

Authentication is no longer bound to a table in the MySQL schema. One of the most interesting authentication plugins is the LDAP plugin, allowing you to use your central LDAP server for authentication rather than maintaining a separate list of credentials within your database server, however this can be complex if you don't already use LDAP in your organisation. For development you might want to use the PAM plugin so you can log in with your usual linux credentials. If you're happy with the MySQL way of doing things, the Schema authentication plugin allows authenticating against credentials stored in the database. Whichever authentication scheme you choose, you'll probably want to disable the "Allow All" plugin which allows unrestricted access.

Console Plugin

With the console plugin enabled you can connect directly to the database, no client required. As this is like most other *nix programs, simply connecting STDIN and STDOUT, this allows for piping input and output directly to and from Drizzle without an intermediary client.

JS Plugin

This plugin adds the JS user function that will run a javascript snippet within a SQL statement. It's got a built in JSON parser so you can store and operate on JSON strings in the database. Both the JSON parser and JS interpreter are implemented by Google's v8 javascript engine. At the moment it's not a first class language within Drizzle, but there are plans to allow it to be used as a language for writing stored procedures.

Memcached

For most applications, caching is a Good Idea (TM). Memcached is commonly used in the same stack as MySQL, and Drizzle has a few tricks up its sleeve to make this easier. The normal access pattern is:

• query the cache
• if a key isn't present in the cache, query the database
• once we've got the information out of the database, push it back into the cache

With the Drizzle memcached_functions plugin, we can send the data off to memcached in the same SQL statement we use to retreive the data. This gives us a couple of interesting possibilities. How about a trigger to prime the cache when data is inserted into Drizzle, or to invalidate cache when rows are deleted? The only slight issue with this is that there isn't a plugin that implements triggers yet, but I'm sure one will be along shortly.

Message Queues

Drizzle provides plugins to publish transactions to both RabbitMQ and ZeroMQ. If you're currently polling your database for new data, this would allow you to move the load from your database to a message queue, providing better scalability. It could also be used as a basis for replication from Drizzle to other data stores.

Query Logging

If the query logging plugin is loaded, by default it logs nothing. This can be controlled at runtime allowing for logging of specific queries or globally at specific times or events. This means that it only logs on demand, and importantly does not require a server restart to turn it on or off. It's also possible to log queries out to a Gearman server for processing. There is at least one person out there using Gearman to integrate Drizzle with MySQL Enterprise Monitor.

Replication

Replication has had a complete overhaul from its MySQL origins. Replication is implemented by storing replication events as Google Protocol Buffer messages in an InnoDB table. Storing replication events in a standardised format in a readily accessible table means that writing replication plugins and clients becomes much easier, simplifying the replication of data between different formats and technologies.

Performance

As I already mentioned, each variant has its strengths and weaknesses. Currently, none of the contenders, including MySQL, offer a complete best, or worst, performance profile for all use cases. If I benchmarked all four databases on identical hardware for various workloads with various levels of concurrency I could easily be here for weeks. So to make everything I've written about features and compatibility relevant by the time you read this, I'm going to give you the distilled highlights and aims for each project. I also can't stress enough that performance is extremely dependant on workload, meaning that the only way to be sure which database server will work best for you is to benchmark it against your application. So let us take a look at the contenders:

Percona

The bulk of Percona Server's performance improvements come from its XtraDB storage engine. Percona identified a situation in MySQL and InnoDB where throughput would steadily increase and then plummet, stalling briefly, before building back up, continually repeating this cycle of build up and stall. XtraDB has made big strides in addressing this problem and will now sustain high throughput, often at higher peak performance, without suffering the same stalling. If your workload requires a high sustained throughput, Percona Server is likely to be a good choice. If your database activity is short bursts of transactions, you're less likely to see such a pronounced improvement.

Another significant improvement is its ability to scale vertically (more CPUs and memory within one server). Percona Server will scale consistently to 48 cores allowing all but the biggest applications to completely avoid the necessity to scale horizontally (more servers using replication and/or sharding), reducing application and administration complexity.

MariaDB

While MariaDB has backported some MySQL 6.0 features and improvements, the current release is based on MySQL 5.1, so there are some advancements that MySQL and Percona Server have that are not yet integrated into MariaDB. At the time of writing there is an alpha release of MariaDB 5.5 available which is based on MySQL 5.5 that will address this. There is criticism about the time it takes for MariaDB to rebase on a new version of MySQL, although I wonder if the project would make its own improvements as quickly if more time was spent in merging in changes.

MariaDB has not been idling in the performance department however, the biggest improvements are in the query optimiser. The role of the query optimiser is to try to run the query in the most efficient manner by calculating the most efficient joins and making the best use of indexes to reduce the amount of disk reads and computations needed to return the result set. Have you ever had to rewrite a subquery into a less readable, more complex join, to make it perform better? MariaDB's query optimiser will do a good job of this under the bonnet. This helps in some cases with writing more readable SQL and also protects your application from a certain class of performance-killing queries that could sneak in from an unwary developer. As well as subquery optimisation, join optimisation has also been improved. Added together, an application that uses lots of complex queries should see good improvements from this. If, however, your application is performing lots of simple queries, it is unlikely you will see the benefit of the improved optimiser.

Instead of using InnoDB, MariaDB makes use of Percona's XtraDB storage engine, so receives similar performance benefits to Percona Server in this area.

Drizzle

Drizzle takes a different approach to performance. Rather than concentrating on raw performance, e.g. how fast Drizzle can process a transaction or number of transactions, they're focused on scaling. Drizzle is targeting workloads where applications need to be able to run 1024+ concurrent connections. MySQL's sweet spot was traditionally around 16 threads, although recent versions appear to cope better with higher concurrency and, as usual, this all depends on workload. From an architecture perspective this means systematically removing contention points within Drizzle that have historically prevented MySQL from scaling. This support for high concurrency workloads has sometimes resulted in design decisions that have caused worse performance at lower concurrency so if your application does not require high levels of concurrency you may not see any performance improvement compared to MySQL and in some cases may actually see worse performance. If you need high concurrency, however, the drizzle is an excellent choice for you.

Summary

The most satisfying conclusion possible to draw from this would be that one of the alternatives offers the best in everything and I could tell you to run out and install it right away. Life is not that simple, however. If you've skipped ahead to peek at the answer, unfortunately this is one of those open ended questions where there is no single right answer.

Each alternative has its advantages and disadvantages, things it does well and areas where others have overtaken it. Which you choose is entirely dependant on on your application workload.

What is abundantly clear though is that MySQL is still going strong after its changes in ownership and this is providing a strong base, and competition, for some interesting projects to develop. It will be interesting to see how they continue to improve and differentiate themselves. It's also impressive to see the community that's developing around them and the level of passion displayed, as evidenced by the comments made around the recent MariaDB benchmarks.

Your Next Step

After doing some initial benchmarking and compatibility testing, the best way of seeing how your application works with one of the alternatives is to run your application against it, however you don't want to migrate everything over until you're sure you've settled on the right one. A good way to do this is to set up a slave that you can point a copy of your application at to perform reads against. You can then configure your application to perform an a/b test, monitoring performance between your MySQL slave(s) versus your alternative slave. May the best alternative win!

                          

There were 34 sessions in all, however a number were repeated across both days. I guess it's hardly surprising as many visitors will attend just one of the two days, and it does give the super-keen amongst us the opportunity to grab more of the talks.

Rasmus Lerdorf gave an entertaining if somewhat unconventional keynote on the Friday - a potted history of the PHP language given by the man himself was, for me at least, very memorable. The second half of the talk concentrated on the new features available in PHP 5.4 and whilst not the usual rousing call-to-arms we expect from a keynote, it was useful to know.

Following on from that was a talk entitled Big Data by James Littlejohn. For me, and a few people I spoke to afterwards, this was a disappointing talk. Littlejohn quite possibly had a very good idea but the talk did not do it much justice. I would have preferred if he had begun by explaining the problem he was trying to solve and then show the software he had built. The remainder of the time could then have been spent looking at the guts of the system.

After that I strolled down to the Unconference and ran into my colleague Marcello Duarte giving a talk about Deliberate Practice which was a nice 20 minute aside and considering how short these talks are, it packed in a good deal of information (or should I say conveyed strong values?). The idea is to practice a task not with the intention of getting it done, but with the intention of getting better at that task. The analogy was drawn from the field of sports and in particular martial arts when students perform drills to increase their proficiency.

Distribute the Workload in the afternoon gave a flying tour on how to build applications that can scale more easily: by using a larger number of smaller servers, making it easier and cheaper to scale the parts of your platform that are under the greatest strain. Of course, more servers means more management overhead and Helgi expertly ran thought the various monitoring, scaling and multi-cloud technologies. I was particularly interested in New Relic for monitoring as it offers a good level of information that you can drill down into for more details. Gearman also got a well-deserved mention for allowing your application to distribute the workload across multiple servers. Interestingly, Gearman tasks can also spawn sub tasks if they see fit, making it a very flexible architecture. If you are about to build a high-availability multi-region application then this was the talk for you.

One of the few non-technical talks was given by June Henriksen. Creative Coding looked at how the brain works and coming from the standpoint that software development is a creative task, looked at how to stimulate the creative parts of the brain to achieve better solutions. For my money the talk would have benefited from a little less emphasis on how your brain works and more practical examples of how to get these benefits in your work environment. It was however, an enjoyable talk, and with a bit of polish this could make a pretty good keynote.

Ian Barber delivered a very interesting session on Teaching Your Machine to Find Fraudsters which explained how you could use SVM (Support Vector Machine), which is now part PHP 5.4, to predict if something is true or not. Ian, who is the maintainer for SVM, put it to good use in this talk and explained how to feed the machine a bunch of historic data about credit card transactions where you know which transactions are fraudulent and then ask the machine to predict if new activity is fraudulent or not.

Friday was rounded off nicely with a panel discussion chaired by Ian Barber with Rasmus Lerdorf, Nikolay Bachiyski and Hugh Williams. The panel answered questions that had been collected via Twitter during the day and further questions fed through as the discussion went on. The general message was that for most of us mere mortals, PHP is more than adequate at scaling. Even Hugh Williams admitted that at one point eBay were considering rewriting their whole front-end in PHP. However, when things get really big (Facebook, Yahoo size) then the problems are not the language but rather the architecture. Even though the discussion was PHP at Scale, much of what was said was relevant to any language. It was a very enjoyable end to a good day of talks.

Last but by no means least was the after-conference social which lasted about an hour or so. All attendees were automatically entered into a prize draw for technical books. At a guess I think they must have handed out 40 or 50 books and it appears to go on for quite a while. I even managed to bag a book myself!

If this years conference had a theme, it was "scaling" with quite a few talks dedicated to this subject. If I had to find a criticism I would probably say that the talks were probably not quite technical enough, but I appreciate you can't please everybody and perhaps the purpose is to whet your appetite rather than provide an in-depth how-to or workshop. Overall a great conference and I'm sure I'll go again next year!

                          

I arrived just in time for the opening keynote, having been quickly ushered in. Nobody even bothered to check that I had a ticket, though I later discovered this wasn't special treatment for me, nobody's eligibility to enter had been checked. I am pleased to report that by Saturday morning they had realised (or been told) that they should be checking attendees were supposed to be entering – a real world example of emergent design perhaps?

Talks - Friday Morning

PHP in 2012 - Rasmus Lerdorf

Rasmus's keynote contained a good insight into the development and motivation behind PHP as a language, and a whistle stop tour of the new features coming to the world in PHP 5.4. It was excellently presented and well received by what was an incredibly packed out room – when I got there I was forced to stand at the back it was so well attended. A rallying call for testing, documenting and further developing the language made me think I should probably try to give something back to the technology that is currently keeping me in a job. The pesky HashDOS problem was explained clearly and I am now wary of json_decode() - I must always remember that users can't be trusted. I was pretty pleased with this conference opener, despite some comments that it was too technical for a keynote, for me it hit just the right note.

Time for a quick cup of coffee and some musings with colleagues before hitting the first of my choices for the main tracks of the conference.

Big Data Web - Big Data Science - Big Data Mining - James Littlejohn

I came into this talk based solely on it's excellent title, I was not alone and the "Big Data" intrigue had gathered a reasonably large crowd. Unfortunately, the content of the presentation seemed to have little correlation to the title, this being simply a walkthrough of what appeared to be a half-written application. My attention waned very early on and never really returned. By the end I still wasn't entirely sure what the point was.

Data Abstraction in Large Web Applications - Brandon Savage

Next up a talk about data abstraction. This was very well presented, conveying a simple idea very clearly. I felt the deliberate avoidance of any particular implementation detail was a good move, leaving just the problem and solution without the specific technical details getting in the way of an important concept. Throwing in the real world example of where it went wrong in his own project was brave, but helped illustrate the point effectively. Maybe the content was a bit basic for the seasoned OO developer, but definitely worthwhile for anyone new to the topic.

Lunch

With not one, not two, but three serving stations, lunch was served pretty efficiently. A choice of lamb pasta or vegetarian lasagne, followed by cake, orange juice and more coffee. Leaving me more than enough time to escape the increasing heat of the room and get outside to take in some fresh air before the afternoon session.

Talks - Friday Afternoon

To a thousand servers and beyond: scaling a massive PHP application - Nikolay Bachiyski

First stop after lunch and the start of what appeared to quickly become a theme at the conference – scaling. Specific details of how wordpress.com scaled up it's operation were presented confidently, but the main talking point after this slot seemed to be the revelation that wordpress.com only have one svn branch (trunk), which is deployed many times a day, with all the in-development code still in it. In some respects it's a shame that this became the lasting impression as lots of the content was interesting. I was particularly interested to see the way wordpress.com has gone about solving MySQL distribution with a tool they have since open sourced (HyperDB). It not only manages read replicas for entire databases, but also individual tables, leaving no requirement for the application code to know where to send queries – an idea that I intend to investigate further.

Distribute the workload - Helgi Þormar Þorbjörnsson

So, following on from a talk about scaling, I then watched another talk about scaling. This time from a slightly more theoretical viewpoint. I thought it was an excellent discussion covering both availability and performance (or more accurately perception of performance) of web applications. The main thing to take away from the approach was liberal use of messaging queues, monitoring and, as the title suggests, distributing the workload to many different components, making for easy horizontal scaling. The talk ran over the allotted time slightly and still I felt Helgi had more to say. I would have been happy to sit and listen to his thoughts for longer too.

Try { Getting People To Come To A Talk About Exceptions } - Ed van Beinum

Another excellently presented topic. A brief tour through the history of the Exception, then some details about the particular PHP quirks. It was definitely worth seeing and the 'Are you prepared to end the program?' question will no doubt enter my mind every time I type the word throw from now on. In fact, I think I'd like my IDE to ask that question of me when I type throw, just in case I forget. That can only lead to better software.

Panel: PHP at Scale - Ian Barber, Rasmus Lerdorf, Nikolay Bachiyski, Hugh Williams

I found this portion of the day interesting to listen to, some good questions were asked, and good answers given. Ian Barber did an excellent job of chairing the discussion, but for future events I would suggest giving each of the panel their own microphone to encourage more debate. The format was good and it was an excellent way to end the formal part of the day. The main thing I came out with was that if PHP is your bottleneck when trying to scale, you're probably not doing it right.

Networking and Raffle (and beer!)

We all know there is no such thing as a free beer, but this is about as close as one can get. To top it off some lucky attendees also got themselves a free book, alas I was not a winner this time. I think this 'event' was actually better suited to the aftermath of a PHP conference than the party that followed it, and I'd encourage after parties to be more in this style for future events, even if I did nip out early to go and grab something to eat in a nearby Spanish restaurant.

Mid Conference Party

Having fed myself I returned to the masses, but ended up only attending this party very briefly. The venue for it was not especially well suited and the loud music made chatting to fellow conference goers difficult. Though anyone who spoke to me on the night will know that this is just a thinly veiled attempt to hide my disgust at being served icy cold stout.

Talks - Saturday Morning

Challenges at scale: extreme data and platforms at eBay - Hugh Williams

So day two arrived, and keen to learn the lesson of arriving just in time on Friday, I arrived in plenty of time to find a good seat for Saturday morning. If I'd known it wasn't going to be quite as well attended and seats would remain free throughout I could have grabbed those extra few vital minutes of sleep. Nevertheless, I was pleased I had made the effort to be there for another splendidly presented keynote. I didn't take away anything I think will be especially useful going forward, but it was enjoyable to listen to and interesting to see some of the infrastructure choices at eBay.

Security audits as an integral part of PHP application development - Sijmen Ruwhof

After a quick coffee break I attended this presentation based on a recommendation from someone who had seen it the previous day. The justification of why security is an important aspect of development was good and the idea of nominating a member of the team to be responsible for the robustness of an application is interesting. There was also a brief introduction to a wide variety of testing tools. However, I felt more time should have been given to explaining the techniques and results. The live demo was not that easy to follow and it would have been better to spend the time clearly explaining the concepts.

Open Source Search: An Analysis - Justin Finkelstein

I came away from my next session slightly disappointed - a description of four open source search platforms with some interesting points, but without a solid comparison. I think I'd have preferred fewer implementation details and more discussion points. The offered conclusion appeared to be that the different platforms are suited to different situations – but after the talk I still have no idea which situations are suited to each. It is possible that I just missed it, since the delivery was very dry and I had begun to think about lunch.

Lunch

In much the same efficient way as Friday, lunch was served up. This time my choice was beef casserole, with the cake, orange juice and coffee thrown in for good measure. Since it wasn't quite so hot today a trip outside wasn't required, so I spent a few minutes at the O'Reilly stand looking at their book selection and wandered upstairs to see the Unconference schedule. I was intrigued by some of the things scheduled at the Unconference in the afternoon, but in the end I stuck to the main tracks and didn't actually attend any of them.

Talks - Saturday Afternoon

MySQL Update - Dave Stokes

I had been looking forward to this presentation, I've been using MySQL now for as long as I've been using PHP and so was hoping to find out what the future holds. Unfortunately, this seemed to be a glorified advert for MySQL Enterprise edition. I've had some experience of the enterprise tools and they can be useful, but this didn't seem to be the right place to discuss them. I had attended the talk with the promise of hearing about the new features of MySQL 5.6, but I don't think I really got that from it.

HTML5 for PHP developers - Andrew Betts

After the disappointment of being advertised to in the previous talk, I really enjoyed listening to this presentation. I felt it was well delivered and it easily held my attention throughout. Although briefly touching on some HTML5 features, the crux of the talk was about solving new problems that the next level of the web will have to overcome. The ideas were good and I shall definitely try to remember the solutions if (or more likely when) I meet the same problems. I was pleased to see a nice balance between discussing both a particular implementation and the underlying concepts that went into the solution.

PHP Under The Hood - Johannes Schlüter

For only the second time across the two days I remained in my seat to see the next presentation in the same room. I attended not really knowing what to expect, or how useful knowing what was happening to my PHP code when it was executing would be. As it turned out I think there were some useful points, along with some not so useful points, but it was enjoyable nonetheless. If I take anything away from this it will be that references in PHP are slower than copies. But in reality, I don't suppose the slow down for using references would really be that noticeable – still, I'm happy to go with a 'more is better' philosophy on this one.

Panel: SQL vs NoSQL - Johannes Schlüter, Helgi Þormar Þorbjörnsson, Derick Rethans, Dave Stokes, David Zuelke

This second panel discussion was highly anticipated, but I don't think much was really covered here. This time round the lack of microphones is probably what made it so entertaining. I felt that once again it was very well chaired, this time by Helgi Þorbjörnsson. I'm not going to complain about the subject matter not being tackled with any great force, since by this time I was flagging a bit anyway and this was a good light-hearted session to finish the conference.

Conclusion

To sum up the conference in one somewhat uninformative metaphor I'd say 'it was a mixed bag'. I thought
the panel discussions worked very well and I would like to see more of them in future. Scaling was definitely the hot topic of this year's conference, and I'm sure it will continue to be a hot topic throughout 2012 - though my current project does have me heavily thinking about scaling so maybe I've just got it on the mind. From what I can tell, PHP's future looks bright, but now is the time we should all be helping out (https://bugs.php.net/random). Some of the subject matter was very introductory, but I still came away feeling I had learned something new and certainly hadn't completely wasted my time. That means the conference, in my book, was a success.

                          

With Twitter moving its API to OAuth the idea of using tokens rather than passwords for authentication went mainstream. Many explanations of OAuth make it seem complicated whereas in reality the "OAuth Dance" is a series of simple steps executed in sequence. This talk covers consuming and providing OAuth services, includes implementation examples, and is recommended for all technical leads, architects, and integration specialists.

You can find Lorna's slides at http://www.slideshare.net/lornajane/oauth-7990565

                          

This year, the conference expands into a new dimension with the addition of a new sister conference, the Dutch Mobile Conference, running alongside DPC. All attendees will get access to both events. The addition of an additional strand to the conference is a response to the situation as we see it in industry. Our clients need mobile solutions alongside their web applications, and developers increasingly work across both web and mobile disciplines.

Call for Papers

The calls for papers for both events are open now and we'd be delighted to hear from you. You don't necessarily need previous speaking experience (but tell us if you have), you just need a great topic - and for these events, the more technical, the better! The URLs you need to submit your talk are http://www.phpconference.nl/call-for-papers and http://www.mobileconference.nl/call-for-papers - we can't wait to see what you come up with, but you need to be quick! The calls for papers close on Tuesday 31st January.

Come to the Conference!

Hopefully you've been enjoying our DPCRadio series, where we share the recordings from last year's PHP conference (if not, you can catch up on the DPCRadio page). This year will be bigger and better than ever, and we are sure you'll love the new mobile elements of the event as well - so come and join us, 7th-9th June in sunny* Amsterdam!

To follow along with the news from both events, follow our twitter accounts @DPCon and @DMConf.

* sunshine not guaranteed

                          

You already know Singleton, Signal/Observer, Factory and friends. But, which object oriented patterns are en vogue in the PHP world and how can you seize their power? This talk gives you an overview on Dependency Injection, Data Mapper and more OO patterns the PHP world talks about right know, using practical code examples.

You can find Toby's slides under "Advanced OOP Patterns" here: http://qafoo.com/presentations.html

                          

As developers, we typically think about the application domain problem and logic and then design our architecture and consequently the database tables from that. We see the database as merely holding the data and our programs as the part that applies logic and processing to that data. We develop our scripts in isolation from the rest of the finished system and rarely think about their impact on other processes and whether we are making good use of what PHP and MySQL are good at. There may sometimes be excellent reasons for this, but I suspect it is more often the case that:

• We are not really aware of what MySQL has to offer, or if we are we are not confident in how to use the features.
• The use of ORM (Object Relational Mapping) is growing and this tends to distance us from the actual database and the SQL. It is perfectly possible now to create an application without writing an SQL query at all.
• We like to keep control in our hands and using some of these features mean the database doing more and our code a lot less which feels threatening.
• We tend to develop in an isolated environment where the only thing running is our script; where we don’t have to worry about competing for resources and speed over a network. Increasingly our live applications are actually running on a network of servers and complex applications need all the help they can get to meet the requirements of page load times and to avoid contention for resources.
• As PHP developers we traditionally create a single web application with a database back-end that is just used for that application so re-use doesn’t seem such an issue. However modular or service-level applications are growing; I’ve worked with several of them in recent years.

If all our programs are doing are simple CRUD (Create, Read, Update and Delete) operations on single tables then just using the most basic features of MySQL is probably sufficient, but if it is more complicated then we could be missing out on features that will really improve the performance and resilience of our applications. We could also save overall development time through re-use of database elements. Our overall aim is to produce fast, reliable, robust applications and we let ourselves and our clients down if we don’t use all the tools at our disposal.

MySQL Tips and Features

So what are these features I’m referring to? I propose to review the options here and then cover a few of them in a bit more detail. I’ll start with the things that some of you may already be using and then move on to the more obscure. As a general rule the aims here are to move less data from database to program, to have each query execute as quickly as possible, and to keep our data as secure as we can.

Fetch a Limited Dataset

Limiting the number of columns you select, limiting the number of rows with LIMIT and careful use of the WHERE clause all reduce the amount of data being transferred. Another useful and often underused option is using the GROUP BY clause along with aggregate functions to read summary data, rather than reading a whole dataset and then performing calculations on it in your program. Imagine you want to get some summary information about a company’s departments and their members of staff. You could produce a query that reads all the staff records and then process that in your program, but a more efficient approach is to get MySQL to do it for you by using the GROUP BY feature.

Link Tables Efficiently

Use joins to read from two or more related tables at once. Inner joins, either explicitly using the JOIN keyword or by listing tables in a comma separated list, get all the records where there is a matching record in each table. Outer joins (LEFT or RIGHT) select all the records from one of the tables and then the values for the subsequent tables if found, or give NULL values if there are no matches. There is also a self join for those occasions when you need to compare the values in different rows of the same table. Joins are a key concept and well worth mastering if you are not already comfortable with them. What is less well known is that joins can also be useful when doing multiple table updates and deletes, although they are used implicitly with a comma separated list in this context rather than with the keyword JOIN.

Sub-queries can also be used in a number of ways to create complex queries that select data from a number of tables. There can be a lot of overlap with joins, in that often the same query can be written using either a join or a sub-query, and a join may well be more efficient. A sub-query can produce a scalar value for use in an expression, a column of single field results, a single row with several fields or a whole table of data that takes the place of a regular table. To find out more the mysql documentation is to be found at http://dev.mysql.com/doc/refman/5.5/en/subqueries.html.

Analyse Your Queries - Can They Be Improved?

Use MySQL's explain keyword when writing your queries to determine the most efficient way to write them when there are alternative ways of doing the same thing. EXPLAIN analyses the query and shows how many table rows have to be read to produce the output and which indices are used. Judicious use of indices can improve performance but poor use of them can actually make things worse, so they are worth giving careful thought to if you have sufficient access to/control of the database. Using EXPLAIN can also help you determine if changing the order of tables in a JOIN may improve the efficiency of a query.

It is very simple to use; when you think you have your query written, just run it with EXPLAIN in front of it. Then try some alternative ways of achieving the same output, or tweak the indices and compare the results.
eg.

EXPLAIN 
SELECT staff.name, department.name, grade 
FROM departments LEFT JOIN staff ON department.id = staff.department_id ;

Use Prepared Statements

Any security article covering PHP and MySQL will cover the desirability of using the prepared statements feature in PDO (PHP Data Objects) to guard against SQL injection. This is where the query is defined with place holders for changeable parameters which are then bound to it at execution. It also improves performance if the query is to be run a number of times with different data, as the query is only parsed when prepared rather than every time it is run as with a normal query repeated. I would view using these wherever possible as mandatory.

Transfer Data Efficiently

LOAD DATA INFILE (and the accompanying mysqlimport command line tool) is the most efficient way to insert a number of records into a table. Next time you need to do some sort of bulk import consider whether it is practical to have an appropriately formatted csv file transferred to the server (perhaps by FTP or equivalent ) and then use LOAD DATA INFILE to process it rather than reading it line-by-line in your script and inserting each record individually. You will have a much faster script that will have less impact on the server.

Enforce Data Consistency with Triggers and without Network Traffic

Triggers are defined statements that are run either before or after INSERT/UPDATE/DELETE operations on a table. They can be quite complex operations with the use of a block of statements and conditions. They can save the programmer having to remember to implement this logic each time the data is used and therefore help to force the integrity of the data. As an example, related records could also be deleted or updated to reflect the change or an entry created in an archive table when a record is deleted. They tend to have a bad reputation amongst developers; I suspect it is because they instinctively don’t like something they aren’t controlling. Perhaps if we thought of them in terms of the observer pattern we would be more at ease with them!
eg.

delimiter |
 
CREATE TRIGGER demo 
  BEFORE INSERT 
  ON t1
  FOR EACH ROW 
  BEGIN
    SET full_name = CONCAT(NEW.first_name, ' ', NEW.last_name);
    DELETE FROM t2 WHERE code = NEW.code;
  END;
|
 
delimiter ;

Pick the Right Database Engine for Your Tables

Database Engines are an important element to consider when designing a database. A default engine is set for a database, but can be changed per table. Think about what sort of table you are defining, and don’t just let MySQL default to the MYISAM engine. MYISAM tables are good for tables largely used for lookup and searches but if it is to be updated frequently the INNODB engine which comes with the option of transactions is a better choice as the use of transactions will improve data integrity.

Take Advantage of MySQL Views in Your Application

Views are stored select statements that can then be used in a query as if it was an ordinary table. They can select data from tables and also other views using most of the constructs available to any SELECT query. The only proviso is that there has to be some solid data behind it, so temporary constructs such as user variables and temporary tables are not available in views. The SELECT statement in the view is actually run when the view is used in a query; either the SELECT criteria merged with those of that query or run first to produce a temporary table which is then queried further by the end query.

Views can also be used to hide table structure changes, such as a table split into two. A view could be created with the old table name and effectively the same data, and then users would be able to query that as before without disruption.

Views particularly come into their own when reports are being generated. The view can be designed to select the base data the report(s) are based on, and then a report query can select and perform calculations based on this set of data - without the writer of that query having to know all the ins and outs of where the data is obtained. If the temporary table option is used then the underlying data tables are not being tied up for longer than necessary and are free for other users. A standard temporary table could be created for the same purposes but that has the disadvantage of being more likely to get out of date and using a view also means that the SQL for creating the "temporary table" is kept in the database and is accessible by all users with the relevant permission. Depending on the exact circumstances, this may or may not be appropriate.

To illustrate a view in action (although there are doubtless better ways of achieving what this does!)

CREATE ALGORITHM = TEMPTABLE VIEW engineering_staff (name, location, grade, manager_id) AS
SELECT CONCAT(staff.firstname, ' ', staff.lastname), department.location, staff.grade, staff.manager_id)
FROM department LEFT JOIN staff on department.id = staff.department_id
WHERE department.name = 'Engineering';
 
SELECT name, location, grade
FROM engineering_staff
WHERE grade > 5
ORDER BY grade;

If there is a clear one-to-one correlation between a view’s output rows and the underlying data then views can also be used in UPDATE and DELETE statements.

Procedures and Functions in MySQL

A few years ago, before joining Ibuildings, I spent some months working on a major application that used PHP with the Oracle database and the application made very heavy use of Oracle's procedures and functions; the SQL in the code itself was largely confined to call statements to use the procedures. This functionality was new to me as it wasn’t available in MySQL at that time (it was introduced with version 5), so MySQL developers don’t have a history of using these in the same way that users of other databases do.

The MySQL documentation defines a stored routine as:

A stored routine is a set of SQL statements that can be stored in the server. Once this has been done, clients don't need to keep reissuing the individual statements but can refer to the stored routine instead.

It is effectively a mini SQL program that is stored in the database; it has inputs, outputs and processing. Stored Routines is a term used to refer to two constructs that have many similarities but also some important differences.

Whilst MySQL provides a wide range of built in functions such as CONCAT() or YEAR(), it is also possible to write your own. Just like stored procedures, they can take input parameters and do some complicated processing but they must return a single value of a declared type. A function is used in an expression within an SQL statement just as the built-in functions are, for example:

SELECT id, name, MYFUNC(grade), dept FROM staff ORDER BY name;

A stored procedure is more complex, it has both input and output parameters and the results of a SELECT statement are returned to the calling code if not part of a cursor, just as with a standard SELECT query. It is actioned by using the CALL statement. Quite simply it would look like this:

CALL my_procedure();

What do Stored Procedures Have to Offer?

In a nutshell:

• You can write considerably more complicated SQL than in a simple query, expressing more complex logic. There are control structures available, such as if/else and loops. With cursors you can iterate through the output of a SELECT statement.
• By doing the logic processing within the database less data needs to be transferred from the database server, improving performance, although it does make more demands on the database server’s memory.
• There is error handling available, so a failure doesn’t have to be a show-stopper, improving robustness.
• The code can be re-used by a range of applications and each developer does not have to re-invent the wheel. With the use of parameters both in and out the procedures can be written to be context dependent, increasing their flexibility and usefulness.
• As with the client/server contract between classes the calling program doesn’t have to know how the database is organised so it is easier to change this without breaking applications or spending a considerable amount of time tracking down where the data is used.
• The procedure can be written to use the privileges of the user who creates it so enabling securer use of sensitive data which can be processed without being returned to the client and without the script’s users needing to have the level of privileges needed to access the data directly.

How to write a stored procedure is a tutorial subject in it's own right, so I don't propose to go into it here. The mysql documentation at http://dev.mysql.com/doc/refman/5.5/en/create-procedure.html is a useful starting point.

There are many different options for using MySQL with PHP; here I will give a quick example of calling a stored procedure from PHP using PDO, which is my preferred method. Suppose we had a stored procedure which takes an input string, does some clever stuff with some tables and then returns a single output value and a set of records from a select within the routine. In PHP you might see something like:

$db = new PDO('mysql:dbname=test;host=localhost', 'testuser', 'testpassword');
$data = $db->query("CALL my_procedure(‘input string’, @output);")
                  ->fetchAll(PDO::FETCH_ASSOC);
$output = $db->query("SELECT @output;")
                       ->fetchAll(PDO::FETCH_ASSOC);

Getting the results of the select will look familiar, but getting the output value is a bit more complicated this makes use of MySQL’s user variables - denoted by @. This can be initialised beforehand with a SET query (SET @output := 0;). However using it without prior initialisation will initialise it to null and then the procedure populates it with a value. To get the contents of a user variable you need to select it.

Making the Most of MySQL

Although this has been a lightening tour, I hope I've given you a taste of the features in MySQL that I'll be using in my applications in the future - and that I've whetted your appetite to find out more about them and to be more adventurous with MySQL. If there's a particular feature that you think belongs on the list, or you can personally recommend a tutorial site, then leave us a comment and tell us what it is and why!

                          

The web is full of useful advice focussed on pushing out the last bit of performance of your code. They mention trivial changes. like changing every occurrence of print with echo even suggesting to use for instead of foreach. These optimisations help, but you are not going to notice it unless they're in a tight loop with many iterations. It is also a wrong approach for tackling performance issues. Before you can optimise, you need to find out if your codeis actually slow; then you need to *understand* the code; and *then* you need to find out where you can optimise it. This talk introduces tools and concepts to optimise the optimisation of your PHP applications.

You can find Derick's talk slides over on his site [PDF]

                          

CouchDB is a prominent representative of the NoSQL movement. Using its integrated web server and eventual consistent replication you can not only distribute data, but also full application code. This even works for clients which are not always connected to the internet, like e.g. mobile devices. This session gives you an insight Couch apps, their beauty and pitfalls.

You can find Kore's slides over on http://talks.qafoo.com/

                          

This session starts with a brief but important overview about the growing Oracle technology eco-system. It shows what Oracle's direction means for PHP application development and deployment.

The majority of the talk then highlights techniques on building high performance PHP applications with the very widely used Oracle Database. Techniques include connection pooling, application monitoring, automatic data privacy for PHP application users, online application upgrades, caching for performance, and how to suspend and resume database transactions to effectively build stateful web applications.

You can find Chris' slides on oracle's technetwork

                          

Architectural and design patterns help software architects to break systems in to smaller, more maintainable sections organised by their functionality and usage. The biggest benefit of patterns is that someone has already solved problems we may face and by utilising patterns such as Transactional Script, Domain Model or Data Mapper in your application it gives us, as developers, some good guidelines on how it should be designed.

Now take a a look at the code in your current project. Is it as well organised as that workshop? Is every class grouped by its purpose, in a nicely organised directory structure? I suspect most of us will answer "just about" to these questions. Most web applications are built of various subsystems, where each one is responsible for different things, such as reading from and writing to a database, retrieving data from a feed, and so on. It's exactly the same as in the workshop, different parts of the system have different purposes; hence they should be grouped in layers. Think about a cake and how it is layered, with sponge at the bottom, jam and cream on the top.

Subsystems, with different responsibilities, are arranged in layers, where each one rests on the layer below it. Higher layers use APIs or services provided by the lower layer, however this lower layer is completely unaware of the fact that there is a layer above it - and it hides another lower placed layer from the layer above. Such architecture encourages separation and decreases coupling between different parts of a system.

There are three major layers in software architecture:

• presentation - the layer contains all the logic responsible for handling interactions between the systems and an user, whether by using GUI or command line. The layer is responsible for displaying information to the user and collecting user input that is passed to domain logic.
• domain - the domain logic layer is also referred as business logic. This layer is the workhorse of an application, it does all the work the application needs to do. The domain logic does calculations on users’ input or stored data. It passes the data to data source layer so it can be persist, furthermore, the layer validates input passed from the presentation layer.
• data source - the layer’s purpose is mainly to communicate with other systems, such as databases, messaging systems and transaction managers. In web applications the major communication will be with different database systems.

Now that we know what types of layers we can find in software architecture, let's have a look at how we can organize domain logic and data sources layers.

Organizing Domain Logic

There are three main design patterns that help us to organize domain logic; these are transaction script, table module and domain model. The following sections will take each of these patterns in turn and examine their suitability for application in various situations.

Transaction Script

The transaction script pattern is the simplest solution for organizing domain logic. There is a collection of scripts that are executed on a server, and every script handles a single procedure that corresponds to a specific request made by a user or another system. The procedure may involve taking user input from the presentation layer, validating it and then storing the data in a database or passing it to external services.

If you have been using PHP for a while, you may remember the way websites were built in the late 90s and around the beginning of the millennium. You had a massive set of scripts, named as guestbook.php, add_entry.php, and view_entry.php, and each of them did one job. In our example scripts, would present all guest book entries, add a new entry to the guest book and display one single entry.

Transaction script is an ideal solution for systems with only a small amount of logic, because it is very easily understood by developers, it has little performance overhead and it performs well with an elementary data source layer when put into action with patterns such as Row Data Gateway or Table Data Gateway. However, you will discover transaction script's disadvantages as soon as the complexity of your domain logic increases. Quite often various transactions must perform similar actions, which leads to duplication of code across them. Some of the code can be refactored into separate classes or functions, however it gets harder and harder to keep the application's structure well-organised as the complexity grows.

Table Module

The next design pattern for organizing domain logic is Table Module. It presents a database-centric approach; all the business logic is organised around database tables. In Table Module, a single class encapsulates all the domain logic for all records stored in a table or view. It's important to note that there is no translation of data between objects and rows, as it happens in Domain Model, hence implementation is relatively simple when compared to the Domain Model pattern (which is next on the list). Table Module provides an interface to perform operations on all the data, so every time there is a need to work on a particular row, a method expects some kind of identity reference to be passed as one of the parameters.

As mentioned before, Table Module is much simpler than Domain Model, making it an ideal middle ground solution between Transaction Script and Domain Model. With Table Module you gain much more structure and lower level of code duplication comparing to Transaction Script, because the domain logic is organized around tables rather than procedures. Table Module is an ideal candidate when an application makes heavy use of tabular data, which normally is a result of a SQL call, because it supports a Record Set pattern very well. However, if your application requires direct instance to instance relationship, then Domain Model will be much more appropriate choice as it better handles complicated domain logic.

Domain Model

The third and more complex solution for organizing domain logic is a design pattern called Domain Model. The pattern provides an object-oriented way of dealing with complicated logic. Instead of having one procedure that handles all business logic for a user action, there are multiple objects and each of them handles a slice of domain logic that is relevant to it. When you start building a model of your application's domain, which in fact is an abstract representation of reality, you will find that initial classes are organised essentially around the nouns in the domain. For instance, when you build an e-commerce solution you will end up with classes such as Customer, Basket, Category and Product; these are the first things you will discuss when thinking about the domain. In a nutshell, each of the classes will incorporate both the data, mainly represented in form of attributes, and all the behaviour that is related to that data. Additionally, Domain Model objects are usually in a one-to-one relationship with records in database tables.

Domain Model is built out of small, loose objects with elegant interfaces, that stay completely free of any external dependencies. Persisting data in Domain Model can be done by implementation of the Data Mapper pattern; this is a great fit, but it will raise the level of complexity. It helps to deal with highly complex business logic in well organised manner, however, it also requires an object-oriented paradigm shift from developers wanting to use it. In fact the Domain Model pattern has a very steep learning curve and the process itself may be very frustrating and time consuming for people new to a rich object model. They can find themselves in a situation where they spend more time jumping from class to class looking for the behaviour they are interested in.

Taking a decision on whether to use Domain Model is highly dependent on the complexity of the business logic. Domain Model handles increasing complexity very well, however, it may take months before developers are fully confident working with the pattern.

Choosing the right solution for domain logic of your application

So far we have looked at three architecture design patterns that will help you to organize domain logic. However, the big question is which pattern you should pick for your application. Unfortunately, there is no straight answer. The decision-making process will be mainly based on judging the complexity of the application's domain logic. The rule will be that you would pick Transaction Script for applications with a lower complexity level and Table Module or Domain Model for applications where the business logic is more demanding. In general you wouldn't use Domain Model pattern for systems with simple domain logic, because the cost of understanding it and complexity of data source exceeds the benefit of this pattern. At the same time you need to consider changing requirements and how they may influence your application. Transaction Script and Table Module will definitely hit a wall much faster than Domain Model when the complexity of the business logic is increased by adding more features. Below is a chart representing relationship between complexity of domain logic and effort needed to introduce new features.

Relationship between effort and complexity for different domain logic architecture design patterns.
Source: Patterns of Enterprise Application Architecture by Martin Fowler.

Remember that even after you take your decision there may be new requirements that could change the application's complexity. In such circumstances it may be worthwhile to upgrade your chosen pattern, for instance refactor Transaction Script to Domain Model. However it won't make sense, in most situations, to go from Domain Model to Transaction Script.

Remember also that there is no rule to say you have to pick one pattern only; you can combine them in your design. It is common to delegate some business logic to Transaction Script and then use Domain Model or Table Module for other domain logic.

Organizing Data Source Layer

The main purpose of the data source layer is to communicate with other systems and to retrieve/persist data. When working with PHP you will find that the data source layer will usually connect to some sort of database. It is desirable, as a good coding practise, to separate database access and all SQL queries from domain logic and place it in separate classes. In most cases classes are organised around the table structure of the database, with each class corresponding to a table. It’s worth mentioning that your choice of data source pattern will be affected by which design solution you chose for the domain logic. Let's have a look at some options:

Gateway

When classes are based on table structure of the database, they act as a gateway to the table, encapsulating the SQL and database connection from any domain logic or presentation. There are two types of gateways: Row Data Gateway and Table Data Gateway.

Row Data Gateway

A class implementing Row Data Gateway pattern contains only data source logic, such as database access and simple conversion from database to in-memory types. An object looks exactly like a record in a table or a view, with each column mapped to a property.

Row Data Gateway should never include any of the business logic. If that's the case then it really an Active Record pattern instead.

Table Data Gateway

An instance of Table Data Gateway handles and encapsulates all the SQL used to access and modify all the rows in a single table or a view. The class usually provides a simple interface that includes multiple methods for finding data based on a condition, as well as some insert, update and delete methods. Each of the methods will map input parameters into SQL queries and then execute them against database connection.

Table Data Gateway always creates a record set when it returns data retrieved from database, even when there was just a simple find by primary key method executed. That makes the pattern very well suited to work with the Table Module pattern.

Editor's Note: Keep reading for code examples including Table Data Gateway.

Data Mapper

Very often, domain objects will have some attributes, for instance collections, that are not part of the database schema. This mismatch requires some sort of mechanism that helps structuring and transferring data between the domain logic and the database. This is exactly what Data Mapper does - it is a layer that separates the business logic from the database by moving data from one to the other. A simple implementation of Data Mapper would just map table columns to equivalent attributes of a domain class.

The great benefit of using Data Mapper is the fact that domain objects don't need to know that there is a database present, making it easy to introduce changes to database structure or domain logic. Additionally, the entire Data Mapper layer can be easily substituted with mocks and stubs for testing purposes.

Data Mapper is mostly used together with Domain Model, and we will go on to look at a code example of using these together.

Patterns in Practise

I would like to present an example showing how you can combine Data Logic and Data Source using design patterns mentioned in the article. I will mainly focus on Domain Model, Data Mapper and Table Data Gateway patterns. For the purpose of this example lets assume that we need to build ecommerce web application.

Domain Model

While analysing requirements you will notice that there are few nouns you talk about repeatedly. It’s your Customer, a Product the customer looks for, and a Basket where the product gets added to by the customer. These three are your domain specific classes. Lets start writing some code:

namespace Store;
 
class Customer
{
    protected $id = 0;
    protected $name;
    protected $address;
}

namespace Store;
class Product
{
    protected $id = 0;
    protected $name;
    protected $description;
    protected $price;
}

namespace Store;
class Basket
{
    protected $id = 0;
 
    /**
     * @var ArrayObject collection of products added to basket
     */
    protected $products;
    protected $customer
}

Each of the above classes will include some business logic specific to its domain. The Customer class may have methods for managing shipping address and financial details, the Product class may have methods that calculate taxes or discounts, and so on . The key is to keep these classes small and compact.

Data Mapper

As mentioned before, Data Mapper is used to move data between layers. For the purpose of this example we will have three database tables customers, products and baskets. Based on our domain model we know that we need to feed data into three objects, so the mappers we create will map data from the customers table to the \Store\Customer class, from products to \Store\Product, and lastly, from baskets to \Store\Basket. Below is a starting point for \Store\Data\Mapper\Customer data mapper; all the other classes will look pretty similar at this early stage.

namespace Store\Data\Mapper;
 
class Customer
{
    /**
     * @var \ArrayObject
     */
    private $identityMap;
 
    /**
     * @var \Store\Data\Gateway
     */
    private $gateway;
 
    /**
     * By using dependency injection it is much easier to unit test your mapper
     *
     * @param \Store\Data\Gateway $gateway
     * @param \ArrayObject $identityMap
     * @return void
     */
    public function __construct(\Store\Data\Gateway $gateway, \ArrayObject $identityMap)
    {
        $this->gateway = $gateway;
        $this->identityMap = $identityMap;
    }
}

As you can see constructor takes two parameters, first is an instance of \Store\Data\Gateway that implements the Table Data Gateway pattern. I decided to use the pattern here because it provides additional separation, however, it is entirely up to you how to handle SQL. The second parameter is instance of ArrayObject that acts as an Identity Map, which is used by the mapper’s find methods to ensure that every object is loaded only once. Lets have a look how it works:

namespace Store\Data\Mapper;
 
class Customer
{
    ...
 
    /**
     * finds customer’s row by primary key
     *
     * @param int $id
     * @return \Store\Customer
     */
    public function find($id)
    {
        if ($this->identityMap->offsetExists($id)) {
            return $this->identityMap->offsetGet($id);
        }
        // By using Table Data Gateway we hide away all SQL
        $data = $this->gateway->find($id)->getFirstItem();
 
        $customer = new \Store\Customer;
 
        // Simple mapping of data from Db to Domain Model
        $customer->setId($id);
        $customer->setName($data['name']);
        $customer->setAddress($data['address']);
 
        // Store customer object in identity map
        $this->identityMap->offsetSet($id, $customer);
 
        // And return Customer object
        return $customer;
    }
}

The first thing the find method does is a check whether data has been already loaded from database. If not, the mapper gets the data by using Table Data Gateway and maps it to the domain object. In my example it's a very simple mapping which links column to attribute, however, you can implement more sophisticated, metadata based mapping should you need it.

Lets have a look at how data is persisted when using Domain Model and Data Mapper.

namespace Store\Data\Mapper;
 
class Customer
{
    ...
 
    /**
     * save
     *
     * @param \Store\Customer $customer
     * @return void
     */
    public function save(\Store\Customer $customer)
    {
        $data = array();
 
        // Simple mapping of domain model to table columns
        $data['name'] = $customer->getName();
        $data['address'] = $customer->getAddress();
 
        if ($customer->getId() > 0) {
            $this->gateway->update($data, $customer->getId());
        } else {
            $this->gateway->insert($data);
        }
    }
}

Data Mapper has a save() method that expects an object of the domain model, in this case \Store\Customer. Once again the mapper prepares data and passes it to the database using a Table Data Gateway pattern. The code here is for Customer only but the concept applies to all domain model objects.

I hope that above example will help you analyse the requirements of your next project, and if you decide that these patterns are a good fit, that you will be able to introduce Domain Model, Data Mapper and Table Data Gateway easily. Are you using these patterns or planning on using them for the next project? Leave us a comment and let us know where they fit in your current or future applications!

                          

Tutorial Day

The conference started on Friday with its first ever tutorial day. I attended the "Security" tutorial by Arne Blankerts as it's very relevant for the project I'm working on at the moment. The talk was very enjoyable, especially because for the first time someone put emphasis not only on the software security aspect, but also on the hardware and the physical access control policy. You can put all your efforts and apply all the best practices to make your software secure, but everything can vanish in a moment if anyone can access your data centre without restrictions. My colleague Marco Lopes has reviewed this tutorial in more detail in his PHPNW11 report.

The second tutorial I attended was "Maintainable Applications in PHP Using Components" by Stuart Herbert. It was his first time delivering this content and I have to say he did it in an awesome way. Despite the network connection problems we had (which delayed the tutorial a bit), we got along very well by the time everyone had their environment setup. Stuart's tutorial was full of hands-on code. This course was not only a lot of fun and laughs, but interesting content too. I'm looking forward to using Phix to create my components repository, maybe pairing its workflow with the chef-based one we are already adopting at Ibuildings. Definitely a thumbs up for Stuart's project.

Main Conference

Ian Barber delivered the PHPNW11 opening keynote - an inspiring talk that gave us all the motivation to get back on track and contribute more to the community.

The day continued with Ben Longden's talk on "REST and the Hypermedia Constraint: A Case Study". Ben showed us how constraints are not necessarily bad, but instead can help you improve the way you implement your software. Talking about REST, he clearly demonstrated how the Hypermedia constraint can be very useful for de-coupling the server, provisioning the information and the client consuming it, reiterating what is REST and what is a RESTful API.

Following this, I was in the mood for a bit of PHP internals - and who better than Derick Rethans to talk about them? The talk title "PHP Extensions, why and what?" as expected wasn't about how to write extensions but about what they are and how they differ from components (and why and when you should prefer them to anything else). A very insightful talk; I'm now eager to write my own extension, with a better understanding of how they can be improved and how conceptually they should be taken into account.

I started my afternoon with Alistair Stead's talk titled "Varnish your PHP application, make it fly!", trying to gather more information and key points to convince the client I'm currently working for that it's time to seriously think about replacing Squid in favour of Varnish. Alistair is probably one of my favourite speakers; he gave a very well presented talk that covered caching concepts, mechanics and how Varnish can make your life easier. The winning point for most of the audience, myself included, was showing some numbers and graphs from a real use case which left people drooling at the idea of getting the same results with a proper caching strategy implementation.

BDD is rapidly taking its place amongst the common list of developers' best practices. Ben Waine's talk titled "Acceptance & Integration Testing Using Behat" brought his experience directly from the Sky Bet company field where Behat is commonly used to coordinate BAs, testers and developers in the most effective way. The talk went through the use of Gherkin, the language Behat uses, to defining its test scenarios down to the Phabric library used to define fixtures "on the fly" - once again using Gherkin tables saving you the pain of dealing with it. It also maintains all the SQL scripts used. All in all, a lightweight, well-presented talk, good for those who are not that much into the BDD concept.

To complete the first day I attended Thijs Feryn's talk "CLI, the other SAPI". Why? Well, as a command line warrior I'm particularly interested in exploring PHP's potential as a command line tool. It emerged during the talk that using PHP from the command line is not just useful for batch processing or scripting, but also for a development tool, testing your models and your code behaviour straight from the command line - this is not to be underestimated. What I enjoyed most about the talk was Thijs' approach: he involved the audience in the talk, testing attendees' knowledge with some simple command line snippets and asking people for the expected results. This was one of my favourite talks; if you have the chance to see it in the future, don't miss it.

Day 2, Sunday, was perhaps not as good as Day 1 content-wise. In the end, I only went to two talks out of three, preferring to spend some time socialising with some of the speakers in the hotel lounge about different geeky topics.

The first talk I went to was "Feeling secure? - notes from the field" by Paul Lemon. As I'd already attended the security tutorial the day before, this was perhaps not so relevant for me. There also wasn't enough time to cover all the topics properly, so the speaker was forced to gloss over the slides rather than talk in detail.

To conclude the conference, I attended "Teaching (mostly) PHP" by Clinton Ingrams. Clinton brought his "in the field" experience about how to mentor and teach students programming, especially PHP programming in this case, presenting his teaching framework. During the presentation we debated the effectiveness of the framework and the taught concepts, as they don't necessarily reflect enterprise reality. The general idea and how he categorised the topics within his teaching module seemed reasonable, as generally keeping concepts simple and easy is important. However, I have to agree with the crowd, some of the examples seemed too simple particularly for final year university students who have already had some programming exposure.

Conclusion

The venue was great, ignoring the WiFi connectivity problems and the queue for the food. This was a very social event from day one, warm and funny with a horde of geeks trying to mix with "normal people" (yes, that can happen if you attend the PHPNW conference, so try not to miss it next year). The best thing about PHP conferences is knowledge-sharing; you won't leave without a hundred different thoughts and ideas of how to do things better. Once again - definitely a good time, both personally and professionally. If you weren't there, you missed out!

If you didn't make it to the conference, the PHPNW11 organisers will be releasing all the main talks from the conference as videos over the next months. Follow @phpnw11 on Twitter to be the first to hear when the videos are up.

                          

Security Tutorial

Beware of the dark side, Luke!

For the morning of the tutorial day at PHPNW11, I decided to attend "Beware of the dark side, Luke!", a security tutorial by Arne Blankerts. While web developers tend to give more emphasis to the security issues directly related with their application (such as XSS, CSRF, SQL injection or Session security), Arne's tutorial was very interesting because it focused also on direct machine access, remote OS access, installed software exploiting, and hardware exploits.

Security is not an absolute concept, it consists of layers. How secure your data is from someone who's trying to obtain it depends on the effort someone is willing to put in versus the time/money you're willing to spend preventing that access.

Local Access

This bring us to the first topic of the tutorial, data centre security. Even if you went all the way to prevent remote attacks, can your hosting provider guarantee that someone won't just be able to enter their data centre, and flee with your disk full of client data?

And consider this - if your data is valuable enough to someone, and you have a backup system powered by your hosting company, what if that system is shared by other clients? Can an attacker get a machine and backup service on the same hosting company as you, and use this machine to try to get data from the backup server?

The point here is that there's always something that someone can do to get to your data. How far you need to go to prevent that depends on whether the data is worth that amount of effort. So, the first step towards security is to decide where to host your application. If you deal with very sensitive data that is of great value, you'll probably want to go with a secure data centre, encrypted disks, non-shared backup servers and all data, sent and received, should be encrypted. It's up to you to decide if you need all of this, none of this, or something in between.

Remote Access

The tutorial then proceeded to remote access security. The first step is to disable password-based SSH login and only allow key-based login. This prevents brute force attacks to get your password. Of course, external root access should also be disabled - if you need root access, just give Sudo capabilities to a standard user, or use su.

Under no circumstances should you use FTP. The FTP protocol is very insecure; not only does it send all your data in plain text over the network, but it also sends your login credentials in plain text. It also uses random port numbers for communication, making it easier for an attacker to take over an existing connection and receive/send information to your server.

Apache and PHP

After this, we started taking a look at security from the Apache and PHP perspective. We discussed why PHP's safe_mode is broken and in fact makes you more vulnerable to security issues than protecting you. For this reason, safe_mode should never be on, and is scheduled to be dropped out of PHP soon. On the other hand open_basedir should always be used.

All software exposing, such as Apache version, PHP version, or even if your server is using Apache and PHP, should be hidden from someone accessing your server. Always set expose_php to off on your php.ini file.

If possible, use php-fpm instead of mod_php. It will give you not only security advantages, but also performance improvements. With mod_php every request will cause Apache to load PHP and all of its modules, even if those requests don't require PHP to prepare the response, as for example static pages, or static images. With php-fpm, a number of PHP processes are always running, ready to reply to Apache requests that happen to need PHP.

Hardware

From here, we moved into hardware issues. Custom hardware issues. Again, this is something that requires your data to have enough value to someone, that they will go all the way to get a piece of hardware into your system. One good example is a modified USB dongle that can be casually dropped on a desk and that eventually someone will plug into their computer, or be social engineered to do it. From this point on, a hacker can have access to that computer.

Web Application Security

Next we moved into the security issues that PHP web developers are more familiar with - the exploits an attacker can run directly into your application.

XSS

We started by taking a look at Cross Site Scripting attacks. XSS lets an attacker inject content into a page, allowing them to add JavaScript or HTML to the page. We looked into the three different XSS kinds of attacks, which are the DOM based, the non-persistent and the persistent.

Some rules of thumb to avoid being vulnerable to XSS attacks, include making sure to filter input for the right context, always escape output contextually, always prefer whitelisting to blacklisting, and never try to read user intentions and repair their input, which sometimes can make an attack even worse.

Sessions

After XSS, we dived into sessions, and possible attacks through sessions. First we looked into some problems with sessions, such as coming from an untrustworthy source, being outdated or belonging to a user other than the one we’re expecting. We also noted that PHP doesn’t validate session id, which means a session id can contain malicious characters or not point to an actual session.

We then looked into session fixation, where a session id may be provided via a link using PHPSESSID, via XSS by setting a cookie or via plugins like Flash which breaks JavaScript security. Ways to prevent session fixation include validating the session id manually (i.e. writing custom code for it), always changing session id with each status change, such as login, change of permissions, or restart of session. Never use the URL to pass a session id, always use cookies and if possible send the HttpOnly header to the browser.

Session Data

We then proceeded to discuss why session data should never be trusted. Unless specifically overridden, PHP stores all session data on the same path. This means that if your application is running in a shared hosting environment, an attacker sharing hosts with you can read and write to your session data, and this can even lead to code execution.

Session Hijacking

The most effective protection against session hijacking is probably fully switching to HTTPS. In case this is not possible, always use different session ids for HTTP and HTTPS.

CSRF

Typical protection against CSRF includes using tokens. But for the tokens to be safe, you must not use predictable seeds for your token. An md5 of time(), for example, is easy to fake. Assuming that there's a session when comparing the token with the one supposedly stored on the session, can allow for a CSRF attack, if you fail to check whether the session really contains a token.

Session Puzzling

This can happen if your application uses the same name for session variables in different contexts.

Captcha

Captchas are hard to read, and can be easily crowdsourced. They're also vulnerable to the same implementation failure mentioned regarding CSRF. If the code validates the provided value against a value that is supposed to be in the session without checking for its contents, an empty attempt to get into the page, without an active session, can pass the captcha validation.

Database

In addition to using prepared statements, always make sure field names, table names and sorting fields don’t come from user input, use associative arrays as whitelists for those situations. Always make sure the user can’t send values other than int when setting limits on the number of results.

Use PDO instead of the MySQL extension.

Passwords

Never store plain text passwords. Use salt when encrypting a password and reapply hashing multiple times. Do a quality check on user-supplied passwords. If possible, advise your users not to use common words or common words with characters replaced by numbers (for example replacing and O by 0 or an A by 4 and so on).

Clickjacking

And finally we took a look at clickjacking and ways to avoid it, such as setting X-FRAME-OPTIONS to deny for no iframe embedding, or setting it to SAMEORIGIN to allow only frames from the same host.

Web Services Tutorial

After lunch, I went to the Web Services tutorial, presented by Lorna Mitchell, one of the organisers of PHPNW11.

We started by defining what a web service is: a way for an application to communicate, allowing for integration with other applications, exposing data and/or functionality, letting you refactor your application without the need to change the way others communicate with it. This allows you to have a common API from where different front ends or even 3rd party applications can get data or functionality.

We then discussed JSON vs XML, and why it is preferable to provide both kinds of output if possible. After that, we defined ‘Heartbeat Method’, and what it is used for. It was established that without documentation, examples, and a help point, a Web Service isn’t helpful at all, as people won’t be able to use it.

We then got into HTTP status codes, HTTP verbs and how to use them correctly and in a RESTful way. Still on the subject of HTTP protocol, Lorna spoke of the correct use of HTTP headers and how web services should be stateless, and all information needed should be on the request. Then we looked into some types of web services such as RESTful, RPC (XML-RPC, JSON-RPC or Soap). We looked in more detail into REST resources and collections. On how hypermedia allows a service to be self-documented, and allows URLs to change because they are provided by the API itself.

After a session of questions and answers, the tutorial day was over and everyone headed over to the pre-conference social at Kro Piccadilly to meet other developers and tech people all around.

Day 1

The first conference day started with an outstanding keynote by Ian Barber, "How to Stand on the Shoulders of Giants". Ian spoke about how great ideas often come from the wish to solve a personal problem and not from the intention of creating something revolutionary. He also spoke about how innovation often builds upon the work of others, and how any of us can one day build something great, if we just keep trying to build the tools that will help us get where we want.

After the keynote, I decided to go to Sebastian Marek’s talk "Magic Behind the Numbers - Software Metrics in Practice". Sebastian spoke about metrics to evaluate the complexity of a software project. He presented ways to look for the complexity and try to reduce it. He spoke of sonar, php_depend, php_cpd, phploc, PHPMD and a lot of other tools that can help one get to the bottom of where complexity lies within their application. I then attended Sebastian Bergmann's talk on "PHP Tester’s Toolbox". Sebastian, the author of PHPUnit, covered various testing tools for PHP, reviewing their pros and cons and comparing them to PHPUnit.

After lunch, I went to see my colleague Alistair Stead's talk on "Varnish your PHP Application, Make it Fly!" in Track 2. Alistair went through Varnish’s configurations details, how the options one takes during the configuration relate to the type of website on which Varnish is being applied. He showed some actual stats on how Varnish can speed up a Magento ecommerce website, and gave some details on how to use both together.

After Alistair's talk, I headed to Track 1 to watch another Ibuildings colleague - Rowan Merewood’s talk on "Estimation or 'How To Dig Your Own Grave'", which ended up being one of the best of PHPNW11. Rowan’s computer was having issues connecting to the projector, and to make things worse, the previous talk in Track 1 went on longer than expected, leaving Rowan even less time to set things up. After a while the audience started being amused by all these attempts to make the laptop work with the projector, but the solution ended up being the switch of Rowan's laptop for Ben Longden’s MacBook Pro. For some more amusement of the audience, Ben's laptop seemed to be unable to show the slides in fullscreen and the remote control to change the slides wasn't working either. At this point, Rowan recruited Ian Barber to manually change the slides through the talk, and it was time to start. His presentation was flawless, a great choice of images, every detail was taken care of, to the point that his clothes matched the slide theme. The talk focused on how to make project estimates, and how to avoid some common pitfalls that often ruin estimates. During the talk, some Twitter users were rating it as one of the best of the conference, which I will have to agree with.

In the evening, there was an open tab at the hotel bar, where the more than 350 participants met for some drinks and some more PHP-related talks. Later, a few of us decided to take a walk around Manchester and get to know the city and some of its iconic locations such as the legendary Factory.

All photographs in this post copyright (c) Stuart Herbert

                          

With Google constantly pushing the customer expectations of searching, is it time to move away from our database full-text search in pursuit of a more targeted platform? Can implementing Solr offer more than an answer to a search? Implementing a search platform isn’t always suitable for all applications, but in this talk we’ll look at identifying the right search solution, choosing the best way to integrate it into our application and exploring all the benefits a search server can offer.

Edit: Paul's slides can be found at http://www.slideshare.net/paulmatthews86/search-with-solr

                          

Martin de Keijzer

Many people use Zend Framework for it's MVC implementation, but it has a lot of hidden gems. Internationalization (i18n) is one of them. We will look how you can create an application that will have the right languages, currencies, dates and times all based on the location of the visiting user. This session will take away a lot of headaches in international projects and will improve the quality in overall.

Edit: You can find Martin's slides on slideshare

                          

Helgi Þormar Þorbjörnsson

APIs are commonly an afterthought, like a hot tub awkwardly attached to a house — a shoehorned approach that produces a suboptimal app with scarce support that lacks documentation. In effect, APIs are the ugly stepchild of the Web.

This is a sad reality that we are faced with, because many companies make their living consuming third-party APIs and mixing in their own data to create amazing and interesting mashups. In the initial phases of development, there is rarely enough money to develop the app and its API. By the time there’s both demand and money, it can be hard to fit an API on top of the architecture in such a way that the whole thing won’t fall over. APIs should be first class citizens of the Web. Inconceivable? Possimpible? Not at all!

In this talk we will dive deeper into why APIs are an afterthought, how we can change that. We will also touch on how that can benefit your product down the line in terms of resource savings and infrastructure efficiency, as well as the impact it will have on your infrastructure.

This talk is inspired by my phpadvent article.

(You can find Helgi's slides over on Slideshare)

                          

PHPSpec is the first ever PHP BDD framework. It is a port of RSpec to PHP created back in 2007 by Pádraic Brady and Travis Swicegood. Development in this framework stopped for a while and was reignited last August (2010). PHPSpec can be installed via pear, using these commands:

$ sudo pear config-set preferred_state beta
$ sudo pear channel-discover pear.phpspec.net
$ sudo pear install --all-deps phpspec/PHPSpec

If you are accustomed to unit testing, here is a quick translation sheet for xUnit/xSpec terms:

• In xUnit we test; in xSpec we describe, so your class names begin with "Describe"
• In PHPSpec, the spec file for a class MyClass is named MyClassSpec.php
• In xUnit we group tests in a TestCase. In xSpec we have examples that are grouped into Contexts
• In xUnit each method is a test with the prefix test. In xSpec each method is an example, we use the prefix it
• In xUnit we assert that something work as expected. In xSpec we specify how it should work

We've used Gherkin and Behat to specify how our application is supposed to work. We use PHPSpec to specify the behaviour of our classes. To get you started, let's see a simple example showing this. We are building a class that greets the user with "Hello, World!". The GreeterSpec.php file would look like this:

class DescribeGreeter extends \PHPSpec\Context
{
    function before()
    {
        $this->greeter = $this->spec(new Greeter);
    }
 
    function itGreetsUsingAHelloWorldMessage()
    {
        $message = $this->greeter->greet();
        $message->should->be('Hello, World!');
    }
}

If you already have experience with Unit Testing, the above would look familiar to you. Note that instead of saying: $this->assertEquals('Hello, World!', $message) we say $message->should->be('Hello, World!'). We are describing how we want the behaviour to be, rather than testing it. The before() method is a setup method that would get run before any example is run; the spec() method is a decorator that wraps the object being tested so we call the expectations on the properties and methods results.

To run the spec we use the phpspec command:

$ phpspec GreeterSpec.php

First we run it and watch it fail. Then we write a Greeter class to satisfy the example.

PHPSpec and MVC

Going back to our Video Renting application. The newly released version (1.2beta) includes integration with Zend Framework which enables us to test the MVC (Model, View, Controller) components individually. When we test an MVC application, we should start with the view, because that's what our scenario describes:

The text "Revolution OS" was not found anywhere in the text of the current page

Let's keep all our specs in a folder called "spec". To write a view spec we first add a folder called views under the spec folder. In our example, this is a view that corresponds to the index action of the review controller. We need to create an IndexSpec.php for our index.phtml view, containing the following code:

namespace Review;
 
// SpecHelper contains the usual ZF bootstrap
// copied from public/index.php
require_once __DIR__ . '/../../SpecHelper.php';
 
use \PHPSpec\Context\Zend\View as ViewContext;
 
class DescribeIndex extends ViewContext
{
    function itRendersTheSelectedVideo()
    {
        $video = \Mockery::mock(
            'Application_Model_Video', array('getName' => 'Revolution OS'));
        $this->assign('video', $video);
        $this->render();
        $this->rendered->should->contain('Revolution OS');
    }
}

Notice that we namespaced our class with the name of the controller. In a modular Zend Framework application you would namespace with the module name and then controller name, e.g. Module\Controller.

If we run the spec we should see an error because we haven't met its description yet.

$ phpspec spec
E
 
Exceptions:
 
  1) Review\Index renders the selected video
     Failure\Exception: $this->runExamples($exampleGroup, $reporter);
     Zend_View_Exception: script 'review/index.phtml' not found in path (/private/var/www/renting/application/views/scripts/)
 
Finished in 0.004656 seconds
1 example, 1 exception

At this stage we need to create the view to deal with the error we saw. In this example, we will use Zend Tool, which creates the controller/action/view all in one go:

$ zf create controller Review
Creating a controller at /private/var/www/renting/application/controllers/ReviewController.php
Creating an index action method in controller Review
Creating a view script for the index action method at 
/private/var/www/renting/application/views/scripts/review/index.phtml
Updating project profile '/private/var/www/renting/.zfproject.xml'

Now instead of an error, our output shows that we have a failure:

$ phpspec spec
F
 
Failures:
 
  1) Review\Index renders the selected video
     Failure\Error: $this->rendered->should->contain('Revolution OS');
     expected to contain 'Revolution OS', found no match (using contain())
     # ./spec/views/review/IndexSpec.php:17
 
Finished in 0.005078 seconds
1 example, 1 failure

This output means that the view now exists, but it's not showing the described text. That's what we expect, since we haven't coded the view yet. Let's add some code to do so:

<?php echo $this->video->getName() ?>

It should now pass.

$ phpspec spec
.
 
Finished in 0.001043 seconds
1 example

That's progress! However Behat is still not happy, because our controller is not setting the model for the view yet. Let us turn our focus to the controller, and start by creating a controllers directory and adding a ReviewControllerSpec.php.

DescribeReviewController must extend \PHPSpec\Context\Zend\Controller. We also need to specify that our controller will send be accessed by POST and will create the model in it. The controller/action will be routed from "/review" and we can add an example to make sure the route work as expected.

 
require_once __DIR__ . '/../SpecHelper.php';
 
class DescribeReviewController extends \PHPSpec\Context\Zend\Controller
{
    function itShouldRouteTheReviewsPageToTheIndexAction()
    {
        $this->routeFor(array('controller' => 'review','action' => 'index'))
             ->should->be('/review');
    }
 
    function itShouldDispatchToTheReviewController()
    {
        $container = new \Yadif_Container(
            new \Zend_Config_Xml(APPLICATION_PATH . "/configs/objects.xml")
        );
 
        $mapper = \Mockery::mock('Application_Model_VideoMapper');
        $mapper->shouldReceive('find')->andReturn($container->videoModel)->with('1')->once();
        $container->videoMapper = $mapper;
        $this->_getZendTest()->bootstrap->getBootstrap()->setContainer($container);
 
        $this->post('/review', array('id' => '1'));
    }
}

We'll also want to add the code for the controller itself. Assuming you are using Yadif or another IoC (Inversion of Control) container, it would look like this:

 
class ReviewController extends Zend_Controller_Action
{
    public function indexAction()
    {
        $this->view->video = $this->getMapper()->find($this->_request->id);
    }
 
    function getMapper()
    {
        $container = $this->getInvokeArg('bootstrap')->getContainer();
        return $container->getComponent('videoMapper');
    }
}

We'll see that this now passes, when we run:

$ phpspec spec
...
 
Finished in 0.005271 seconds
3 examples

This looks much healthier, but if we run Behat, it will still be unhappy. We need to pass real data to the view, and then Behat will point out that the next step now is to describe the model. We need to have the data of our selected movies so we can fetch our view properly. Testing the models should be focused on behaviour, rather than the mechanics of database access. We know Zend_Db works, we need to know if we have made mistakes in our model. We will therefore test for the validation, filtering and business rules that we keep in the model.

class DescribeVideo extends \PHPSpec\Context
{
    function before()
    {
        $this->video = $this->spec(new Application_Model_Video);
    }
 
    function itIsNotValidWithoutTheName()
    {
        $this->video->setName('');
        $this->video->shouldNot->beValid();
    }
}

We need to implement a isValid() method in the Video model. PHPSpec does not have a beValid matcher, but it will use predicate matchers and find a method isValid() by magic. Once the isValid() is implemented properly than the spec passes. You can do something similar if the form validation is stored in your (zend) forms. Also note the before() method that will be called before any example is run.

You can describe your mapper's behaviour by inspecting that it calls the Data Access Object (DAO) to fetch to or persist data from the models. At that point you can then add the DAO (e.g. DbTable, Rss, etc).

Should we Hit the Database?

In many cases you simply need to verify the business logic and not the database operation. Hitting the database consumes both time and resources, which will slow down the execution. There will be times where we need to expose some database behaviour or just feel more confident that our models work. In those cases we can use a Test Data Builder pattern (this is a topic on its own, and I will save it for another post).

If you run Behat again after the model specs pass, the view should be displaying the correct data, so your scenario should pass, and a new scenario will be failing. We use Behat to tell us what to do next, and so the outside-in cycle is begins again.

Final Thoughts

In BDD, developers are driven by the specification, rather than tests of things they haven’t written yet. That said, the goals of both TDD and BDD are basically the same: making sure the user gets what they want. BDD makes that focus more explicit, and uses a language that invites the user to write their tests as a specification.

Behat and PHPSpec sit at different levels in the Outside-in cycle. Behat provides the outermost layer, allowing the stakeholders and developers to collaborate. PHPSpec provides the inner layer, allowing the specification of how the classes will collaborate with each other.