You’re using memcached to, among other things, cache information from a MySQL database.

Problem:

There are many clients who read and write to this database but only some of them use memcached. Therefore, not everyone can keep the cache sync’d when updating database values. Those that do use memcached need a current view of certain data, that is, they will react poorly to stale cache. Anndddd…. setting the cache record expirations low enough to solve the “staleness problem” defeats the benefit of caching in the first place (we’re back to lots of DB chatter).

Solution:

Install Patrick Galbraith and Brian Aker’s memcached UDFs for MySQL and set up exploitive triggers to keep the cache both warm and current… downright hot hot hot! *ksssss*

Quick HOW-TO:

What you’ll need:

• The latest libmemcached.
• The latest memcached functions for MySQL.
• Actually, that’s it since I’m assuming you already have MySQL, memcached, some memcached clients, and the basic knowledge of how they work… otherwise you wouldn’t have my problem.

Build and install libmemcached:

This is just standard operating procedure for installing a dynamically linked library.

Build and install the memcached MySQL UDF:

If you’ve never installed a MySQL UDF before, this could require some reading. The README in the package is good but if you want an excerpt just to get it installed quickly as to get on with this how-to:

To build:

./configure \
  --with-mysql=/usr/local/mysql/bin/mysql_config \
  --libdir=/usr/local/mysql/lib/mysql/
make
make install

Please keep in mind that for your UDF to be loaded, it must be in the
library path for your server (and yes, we should fix this). On Linux you can
set this by exporting the correct path in LD_LIBRARY_PATH for your mysql
server.

Set up the basic UDFs memc_servers_set(), memc_set(), memc_get(), and memc_delete() in MySQL:

CREATE FUNCTION memc_servers_set RETURNS INT SONAME "libmemcached_functions_mysql.so";
CREATE FUNCTION memc_set RETURNS INT SONAME "libmemcached_functions_mysql.so";
CREATE FUNCTION memc_get RETURNS STRING SONAME "libmemcached_functions_mysql.so";
CREATE FUNCTION memc_delete RETURNS INT SONAME "libmemcached_functions_mysql.so";

These are the only functions you’ll need for this how-to but you might want to load them all (see the README).

Create triggers to update your cache when particular tables are updated:

So far we’ve only been doing prep. work. Now for the fun part…

For this example, I’ve set up five memcached servers on my local machine on TCP and UDP ports 11411 - 11415. Now, let’s pretend that I want to access important_data for username in the cache_test table by the memcached key “imp:<username>”. I want the cache to be refreshed any time a row is inserted or updated and I want the key removed when the related row is deleted from the database.

To create the table we’re going to play with:

CREATE TABLE cache_test (
  username VARCHAR( 32 ) NOT NULL,
  important_data VARCHAR( 255 ),
  PRIMARY KEY ( username )
) ENGINE=MyISAM;

To set the memcached servers we want to use (this only needs to be set once):

SELECT memc_servers_set( '127.0.0.1:11411,127.0.0.1:11412,127.0.0.1:11413,127.0.0.1:11414,127.0.0.1:11415' );

To create the related triggers, your user will need the TRIGGER and/or the SUPER privileges depending on the version of MySQL you’re running. You can’t create a trigger that returns a result, but there’s nothing stopping you from just storing the result somewhere else. Here are the statements to create the triggers:

DELIMITER |

CREATE TRIGGER cache_test_warmer_insert
  AFTER INSERT ON cache_test
  FOR EACH ROW BEGIN
    SET @memc_rc:= memc_set( CONCAT( "imp:", NEW.username ), NEW.important_data );
  END; |

CREATE TRIGGER cache_test_warmer_update
  AFTER UPDATE ON cache_test
  FOR EACH ROW BEGIN
    SET @memc_rc:= memc_set( CONCAT( "imp:", NEW.username ), NEW.important_data );
  END; |

CREATE TRIGGER cache_test_cooler
  AFTER DELETE ON cache_test
  FOR EACH ROW BEGIN
    SET @memc_rc:= memc_delete( CONCAT( "imp:", OLD.username ) );
  END; |

DELIMITER ;

That’s it. Let’s test it out! Let’s insert some records into the table:

mysql>
mysql> INSERT INTO cache_test ( username, important_data )
    -> VALUES
    -> ( 'zaphod', 'no ego' ),
    -> ( 'charles', 'hides money under his bed' ),
    -> ( 'zilton', 'drinks kool-aid with no sugar' );
Query OK, 3 rows affected (0.01 sec)
Records: 3  Duplicates: 0  Warnings: 0

mysql>

… and then check the cache:

(caylward@barghest):~/local/libmemcached/bin$ ./memcat imp:zilton
drinks kool-aid with no sugar
(caylward@barghest):~/local/libmemcached/bin$ ./memcat imp:charles
hides money under his bed
(caylward@barghest):~/local/libmemcached/bin$

Now that hopefully you believe that this is actually working, the rest of the example will just use the MySQL client:

mysql> UPDATE cache_test SET important_data = "can't write for beans"
    -> WHERE username = 'charles';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql>
mysql> SELECT memc_get( 'imp:charles' );
+---------------------------+
| memc_get( 'imp:charles' ) |
+---------------------------+
| can't write for beans     |
+---------------------------+
1 row in set (0.00 sec)

mysql>
mysql> DELETE FROM cache_test WHERE username = 'charles';
Query OK, 1 row affected (0.00 sec)

mysql>
mysql> SELECT memc_get( 'imp:charles' );
+---------------------------+
| memc_get( 'imp:charles' ) |
+---------------------------+
| NULL                      |
+---------------------------+
1 row in set (0.00 sec)

There ya go. Piping hot memcached. :)

There are however a few caveats to all of this:

• This is all basically using alpha software so normal paranoia applies here.
• The list of memcached servers to use needs to be set any time the database is restarted.
• Trying to change the list of memcached servers more than once should be possible but does not behave correctly (possible bug I’m looking into).

Good luck!

End of transmission.

Basically, error messages are only to be seen by software developers. Exceptions should be reserved for a very small number of situations. One is field (form) validation. Someone was supposed to enter a credit card number but didn’t. An email address was required but something that doesn’t resemble one was provided. In other words, a key element in some transaction. Another appropriate time is during absolute errors when you simply cannot complete a task like when the network is down (but even then there are graceful ways to handle that, like queuing up the operation… offline operating modes, etc).

It should never happen when the user is navigating your user interface — period(.) Even when your user is trying to complete an action that requires more information on their part, it should not happen. When you are down in the crux of some code, it’s easy to get into the habit of thinking in terms of exceptions / errors when you don’t have enough information, but you should never throw the equivalent of these at your end users. If you are writing a user interface and you feel like it’s time for you to bring up an error message dialog box / widget / whatever, stop. Chances are it’s the wrong thing to do. No, really, you need to take a step back.

Ask yourself, is this message basically telling the user “you didn’t do it right, try again”? It doesn’t matter what the user was trying to do, telling them they aren’t doing it right is nothing short of infuriating. (Has anyone here tried using Windows?) More often than not, wherever the user is in your interface, there is enough contextual information for you to help the user do the right thing without them even knowing they are “doing something wrong”.

Example: If I am writing a text message on my cell phone and hit the button for “send”, and I haven’t entered a recipient, I should not get an error message telling me that I need to enter a recipient. It’s clear to every turnip using a cell phone that messages need at least one recipient. My intentions are clear. What should you do to help me send this message? Probably bring up my list of contacts. Don’t tell me that the way I’m attempting to do something simple like send a text message is wrong.

It’s pretty clear in the example above which is the better solution. If you analyze your interface in this way (what’s the user’s intention when I feel that I should be writing an error message? what do they still need to do to accomplish it?) and figure out how to help users instead of insult them, they will love you for it. If you try this method and it seems that you’re getting lost in a never ending matrix of what input is still needed and what isn’t, then you probably need to redesign the whole interface.

Ok, that may be a bit harsh. Perhaps forcefully move the user through a series of steps rather than giving them a whole bunch of ways to do something where some paths are wrong. I would rather have fewer choices on an interface if it means the interface doesn’t need to yell at me. If it still seems too complicated, it might mean that you have picked the wrong organizational units… your grouping the wrong items together… or your trees don’t have the right roots.

The example I gave above happens to be common on cell phones. Instead of having a big swamp of text fields and check boxes for each outgoing / incoming message and then still have to code a text message folder navigation menu anyway, they could instead start out by binding text messages to contacts directly in say a thread format (or even just sequentially). It’s a phone! So most things are related to communicating with others, right? So, why not bind all communiqué to or from the contact in question… to the contact directly from the get go?

(I started writing this before the iPhone came out but didn’t post it until afterwards. Again, because Apple gets this, it’s why people drool over an Apple iPhone but not over a Samsung SGH-AXYZ-123 or whatever)

Don’t tell your users that they are doing it wrong. If you find yourself writing code that tells them so, take a step back and figure out what it is that you are doing wrong.

What happens when a connection dies?
This one is mostly the same as the one above except you might have been half way through an action. So, are you using transactions? If the datasource isn’t transactional, are you making sure to clean up and start over? Continue where you left off?

What happens if the network is simply very slow or has high latency?
This one can be tricky as it’s not an easily recognized error condition or might not be one at all, but one should think about it.

How is redundancy / replication handled?
These ones are closer to things that would be put in the requirements in the first place. I’d argue that these should always be in ones mind however. Always plan for wild success, right? ;) Can it be redundant by adding more servers? Simply, can multiple instances run on different machines? Where’s state stored? How is state shared? Pretty easy questions to answer if you’re writing a web app on, say, Tomcat.

How is load balanced?
This ties in with the one above. Always assume that someone will actually want to use your software (right? hah!) and therefore scale their deployment of it. Use round-robin DNS (generally weak)? Use DNS and iterate over records yourself based on some criteria? Implement DNS SRV records to do that? Go big and buy load balancers?

Is it easy to upgrade?
Usually, if you’ve answered the redundancy / replication questions, a rolling upgrade is easy to plan.

How is the configuration stored?
Are they separate and centralized? Separate meaning do different components have their own configs or share one big one? Centralized meaning are they located in the same spot in the same manner? When is it read? Can it be gracefully reloaded without restarting the service? Is it validated fully before being loaded or acted upon? This doesn’t just mean that the formatting is correct or that each field has a value. Input constraints apply here too… is that integer in the right range? Does that string have an invalid character? Is it stored securely?

How do you log?
Ok… this doesn’t have to do with network software exactly but I’m throwing it on here too. Files? Syslog facilities? Do you have well defined logging levels? This one gets skipped a lot because we all feel that we have a pretty good idea about what’s a debug, fine, info, config, warning, or critical message. The problem is that each person has a different versions of that list. Make sure everyone on the project is using the same scheme. It can be kept simple. Are log entries meaningful to those not familiar with the code or the protocols used? Developers are often not the people eventually running the software. An operations group at another company or different department might be. Even with effort, this one is hard to get right… just ask around if the messages make sense. Maybe ask someone from HR or marketing… :)

Is input always validated for all methods?
Constraints! Have them… and check them! This one is also pretty general. Did one of those pesky null pointers get passed over? What happens if someone passes in a negative number and that doesn’t make sense? Is there only a certain range that makes sense? These checks sometimes don’t get done for fear of introducing “magic numbers” into your code. Everyone hates magic numbers in code but sometimes they are necessary… just make sure they are used wisely and commented. An example might be a method that creates a pool of connections. The number of connections in the pool is passed in as an integer variable. Someone might want to create a huge pool of connections later down the road for some weird reason… but it probably never makes sense to go completely unchecked and let the value 4,294,967,295 get passed in… or -1 for that matter.