define('WP_DEBUG', true);

If you are experiencing the same issue as I do,  you will see an error similar to follow:

Fatal error: Internal Zend error - Missing class information for in .../wp-content/plugins/wp-super-cache/wp-cache-base.php on line 5

Here is the content of this file wp-cache-base.php:

Line 5 points to “class CacheMeta {“.  What it looks like is that APC can’t cope with how this class is dynamically declared.  A workaround is add the following line in your php.ini file.

apc.filters = wp-cache-base

This tells APC to ignore this file so instead of caching the runtime, it gets compiled every time it’s called upon.  The performance penalty is negligible for this small file so it’s safe to use this workaround.

Restart PHP-FPM (or PHP-CGI) now.  The blank pages issue should be gone.  

Good luck! 

First of all, check out the trunk from MacPorts, compile and install.
% mkdir -p /opt/mports
% cd /opt/mports
% svn checkout http://svn.macports.org/repository/macports/trunk
% cd /opt/mports/trunk/base
% ./configure --enable-readline
% make
% sudo make install
% make distclean


Then, update MacPorts:
# sudo port selfudpate
# sudo port upgrade outdated


You may still encounter some dependency issues. But at this stage, it should be very easy to resolve by temporarily pulling old libraries from Time Machine. For example, I ran into issue with gettext. The Lion upgrade removed this library /opt/local/lib/libintl.8.dylib that was used by gettext but it did not remove gettext completely. MacPorts was confused and refused to reinstall gettext. I had to go back to Time Machine to restore this file temporarily so I can set up gettext again. If you don’t have Time Machine, it would be difficult to acquire these old libraries. If it happens that you need this file, you can download it here.

You will need to force activate the module after you placed the library.
port -f activate gettext

Now, you can uninstall or upgrade the modules.

If you are still encountering dependency issues, try adding/removing binpath in /opt/local/etc/macports/macports.conf.
binpath /bin:/sbin:/usr/bin:/usr/sbin:/opt/local/bin:/opt/local/sbin:/usr/X11R6/bin

Some modules may need this binpath in order to install but most do not require this change.

strace -t -p 

You may see a bunch of open calls, such as following, on your CD/DVD drive.

open("/dev/hdc", O_RDONLY|O_NONBLOCK|O_EXCL|O_LARGEFILE) = -1
EBUSY (Device or resource busy)

So, what’s the solution?  I haven’t found a good solution for it other than stopping this pulling entirely.  Usually, you only use the CD/DVD during OS installation so stopping it shouldn’t create any adverse effect on your environment.  Here is what you should do to stop it:

• Make sure haldaemon is still running, run following command to find your CD/DVD drive’s UDI

hal-find-by-capability --capability storage.cdrom

• You should see something like this

/org/freedesktop/Hal/devices/storage_serial_QM00003

• Now create this file, /etc/hal/fdi/information/media-check-disable-cd.fdi, with following content:

  
    /org/freedesktop/Hal/devices/storage_serial_QM00003">
      false
    
    

And of course, replace the “info.udi” value with your own value.  Restart haldaemon now and you should be all good.

Couple ways to do this hack.  One is to modify BigNestedIntArray.MAX_ITEMS directly with your desire max value.  Another way is to modify the following 3 files.

• MultiValueFacetDataCache.java
• MultiValueFacetHandler.java
• BigNestedIntArray.java

Look for the following code:

_maxItems = Math.min(maxItems, BigNestedIntArray.MAX_ITEMS);

and replace it with:

_maxItems = maxItems;

And of course, call setMaxItems on your MultiValueFacetHandler to set the desired max value.

Just want to iterate that this hack has only been tested in one particular project.  It may create instability in your project if you have a lot of records with 1024+ facet values.  I was told that the array’s key is 11 bit.  So, there is a max value at 2048.  Having more than 2048 value will likely cause an exception.  Memory consumption can also become a real issue.  Consider yourself warned.

chcon -R user_u:object_r:httpd_sys_content_t 

You may want to check on /var/www first to see if the context is correct by issuing this command:

ls -al --context /var/www

Okay, this is it.  This should fix this annoying issue.

The biggest roadblock we faced with this implementation is the memory footprint.  When the author index was loaded using Bobo, it allocated 12G of memory.  Initially, we set our young generation size way too small, the GC algorithm we selected, CMS (Concurrent Mark Sweep), had to constantly do full sweep after every 2-3 searches.  The full sweep would halt the entire service for about a minute before returning.  It was unacceptable as it pretty much killed search altogether.  It appeared that Bobo allocates quite a bit of temporary memory to calculate the facets.  Perhaps it was the nature of our data with a lot of intersection between authors that caused the excessive memory usage.  We slowly increased the young generation size from 2G (yes I know, it’s very small) to around 8G to get a stable system with virtually zero full sweep.

Here is our current JVM config:

java -verbosegc -XX:+PrintGCDetails \
     -XX:+UseConcMarkSweepGC \
     -XX:+CMSIncrementalMode \
     -XX:+CMSIncrementalPacing \
     -XX:+UseParNewGC \
     -XX:+CMSParallelRemarkEnabled \
     -XX:+DisableExplicitGC \
     -XX:MaxGCPauseMillis=2000 \
     -XX:SoftRefLRUPolicyMSPerMB=1 \
     -XX:CMSIncrementalDutyCycleMin=10 \
     -XX:CMSIncrementalDutyCycle=50 \
     -XX:ParallelGCThreads=8 \
     -XX:GCTimeRatio=10 \
     -Xmn8g \
     -Xms22g \
     -Xmx22g

This configuration works for us.  If you run into similar JVM garbage collection issue, I hope this set of configuration will help you too.

Join the forum discussion on this post - (1) Posts

Let’s look at an books inventory example:

Author
---------------------------
auth_id | author_name
--------+------------------
      1 | J.K. Rowling
      2 | Michael Crichton
      3 | J. R. R. Tolkien

Category
------------------
cat_id | category
-------+----------
     1 | Fantasy
     2 | Sci-Fi

Book
--------------------------------------------------------------------------------
id | auth_id | cat_id | release_date | title
---+---------+--------+--------------+------------------------------------------
 1 |       1 |      1 |   1999-09-01 | H. P. and the Sorcerer’s Stone
 2 |       1 |      1 |   2004-08-01 | H. P. and the Order of the Phoenix
 3 |       2 |      2 |   1991-03-01 | Jurassic Park
 4 |       2 |      2 |   2000-10-01 | Timeline
 5 |       3 |      1 |   1990-04-01 | LOTR, the Fellowship of the Ring
 6 |       3 |      1 |   1990-04-01 | LOTR, the Two Towers
 7 |       3 |      1 |   1990-04-01 | LOTR, Return of the King

It should be obvious to you that the base record in the above example is Book.  We will denormalize the above tables into a flat structure.  The final table to be fed into Solr will look like this:

-------------------------------------------------------------------------------------
id | author_name      | category | release_date | title
---+------------------+----------+--------------+------------------------------------
 1 | J.K. Rowling     | Fantasy  |   1999-09-01 | H. P. and the Sorcerer’s Stone
 2 | J.K. Rowling     | Fantasy  |   2004-08-01 | H. P. and the Order of the Phoenix
 3 | Michael Crichton | Sci-Fi   |   1991-03-01 | Jurassic Park
 4 | Michael Crichton | Sci-Fi   |   2000-10-01 | Timeline
 5 | J. R. R. Tolkien | Fantasy  |   1990-04-01 | LOTR, the Fellowship of the Ring
 6 | J. R. R. Tolkien | Fantasy  |   1990-04-01 | LOTR, the Two Towers
 7 | J. R. R. Tolkien | Fantasy  |   1990-04-01 | LOTR, Return of the King

Notice that the flatten records look more verbose now.  The redundant data is intended for Solr to build facets from.  Let’s create facets on author_name and category.  First, enable indexing on author_name and category.  Second, enable facet search in the requestHandler.  Here is a sample configuration:

author_name
category

true

100

1

After you rebuilt your index and restarted Solr, you should start seeing facets in your search results.  This is a very simple example of flattening records for faceted search.  As you add more record types to the index, things can get a little messy.  Assuming we have done so well with book search that we want to add music search as well.  Let’s flatten music records using technique we just discussed.  Here is the flatten version of music records:

-------------------------------------------------------------------
id | artist_name      | genre         | release_date | album_title
---+------------------+---------------+--------------+-------------
 1 | The Beatles      | Classic Rock  |   1969-09-26 | Abbey Road
 2 | The Doors        | Classic Rock  |   1970-08-01 | The Doors
 3 | Madonna          | Pop           |   1998-03-03 | Ray of Light
 4 | Prince           | Sci-Fi        |   1990-10-17 | Purple Rain

Notice that this table look awfully similar to the flattened version of the book records.  We can in fact reuse fields to map this new music data so a common search query can search on both book and music records.  However, we cannot just simply mix the two tables together.  That’s because doing so will lose the data type (book, music) information once the tables is merged.  We will have to introduce a new field for record type identification.  We also need to create a globally unique identifier for these records.  Here is what the final merged table look like:

--------------------------------------------------------------------------------------------------------
guid | id | type  | author_name      | category      | release_date | title
-----+----+-------+------------------+---------------+--------------+-----------------------------------
B_1  |  1 | Book  | J.K. Rowling     | Fantasy       |   1999-09-01 | H. P. and the Sorcerer’s Stone
B_2  |  2 | Book  | J.K. Rowling     | Fantasy       |   2004-08-01 | H. P. and the Order of the Phoenix
B_3  |  3 | Book  | Michael Crichton | Sci-Fi        |   1991-03-01 | Jurassic Park
B_4  |  4 | Book  | Michael Crichton | Sci-Fi        |   2000-10-01 | Timeline
B_5  |  5 | Book  | J. R. R. Tolkien | Fantasy       |   1990-04-01 | LOTR, the Fellowship of the Ring
B_6  |  6 | Book  | J. R. R. Tolkien | Fantasy       |   1990-04-01 | LOTR, the Two Towers
B_7  |  7 | Book  | J. R. R. Tolkien | Fantasy       |   1990-04-01 | LOTR, Return of the King
M_1  |  1 | Music | The Beatles      | Classic Rock  |   1969-09-26 | Abbey Road
M_2  |  2 | Music | The Doors        | Classic Rock  |   1970-08-01 | The Doors
M_3  |  3 | Music | Madonna          | Pop           |   1998-03-03 | Ray of Light
M_4  |  4 | Music | Prince           | Sci-Fi        |   1990-10-17 | Purple Rain

As you can see, when we reference more tables, the bigger the flattened table will become.  The table itself also become more human readable because you no longer have to refer to other tables to retrieve the textual representation of a key.  This is exactly what Lucene (underlying search technology of Solr) needs.  Lucene doesn’t care about normalized data structure.  Lucene only care about building indexes on top of flat records and you search against those indexes.  It’s as simple as that.  Lucene is not a database.

There are limitation to this denormalization approach though as we deal with more complex data structure such as multivalued fields.  Let’s say we want to add some sellers and their location information to our search engine where a seller can sell many books/music and the same book/music can be sold by many sellers (This is essentially a many-to-many relationship).  We also want to enable faceting on seller and seller location so user can filter by sellers and their location to locate books/music availability at certain location.  The records would look something like this when flatten.

-------------------------------------------------------------------------------------------------
guid | id | type | author_name  | category | seller          | seller_location    | title
-----+----+------+--------------+----------+-----------------+--------------------+--------------
B_1  |  1 | Book | J.K. Rowling | Fantasy  | Smith, Johnson  | Boston, New York   | ..Sorcerer..
B_2  |  2 | Book | J.K. Rowling | Fantasy  | Smith, Williams | Boston, Providence | ..Phoenix
B_3  |  3 | Book | M. Crichton  | Sci-Fi   | Smith, Wilson   | Boston, Austin     | Jurassic..
B_4  |  4 | Book | M. Crichton  | Sci-Fi   | Smith, Johnson  | Boston, New York   | Timeline

Notice that each book has 2 sellers and 2 seller locations.  Look more carefully and you will see that the relationship between seller and seller location is not there anymore.  We can use their position as a hint to map between seller and seller location but position is not very reliable so this piece of information is considered lost.  Besides, the facet values will be incorrect when user start filtering by seller or seller location.  Here is an example to illustrate the point:

• Apply seller -> Smith as filter
• seller_location facet values would return Boston, New York, Providence, Austin

From Solr/Lucene’s point of view, this is a correct result because it’s basically returning possible seller location values based on records found by the filter.  From user’s point of view, the results is totaly wrong, because seller Smith does not have locations in New York, Providence or Austin.

So, how do we fix this?  Remember I mentioned earlier that Lucene is not a database.  It really isn’t, so a join query is out of the question.  Normally, when we deal with such query, we would do a inner join between seller and book tables and filter on the seller field.  The results will be a list of books with the associating seller location.  In this case, seller_location should return Boston on every records.  It is apparent that the only way to perform this query is by keeping the book and seller tables separate.  To make this query possible in Solr/Lucene, we need to workaround Lucene’s limitation.  We will need additional record type and we will need to run multiple queries.  First, we need to keep the seller table separate (as separate records) because it let us maintain relationship between seller and seller_location.  Second, we need additional column in seller records to map to book/music records to make filtering book/music by seller possible.  The seller records would look like this:

---------------------------------------------------------------------
guid | id | type   | seller   | seller_location | product_guid
-----+----+--------+----------+-----------------+--------------------
S_1  |  1 | Seller | Smith    | Boston          | B_1, B_2, B_3, B_4
S_2  |  2 | Seller | Williams | Providence      | B_2
S_3  |  3 | Seller | Wilson   | Austin          | B_3
S_4  |  4 | Seller | Johnson  | New York        | B_4

For this query to work, we will need to query Solr twice.  The first query will filter on seller -> Smith, type -> Seller.  The facet values on seller_location should return Boston only.  Keep this facet values so you can display it as part of the results.  Then, we need to find all the facet values in product_guid (assuming facet is already enabled on this field) to compose the second query to query against Book and Music records.  The second query will return all associating Book and Music records of seller Smith.  Now you have the right Book and Music records and correct facet value on seller_location.

Solr/Lucene may not be the best solution when dealing with many-to-many relationships.  A database would do a much better job at joining tables.  If it becomes unavoidable and you can only use Solr/Lucene, give this workaround a try and let me know how things go.  If you have a better solution than this, let me know as well.  I like to hear how people solve similar challenge when using Solr/Lucene.

Join the forum discussion on this post - (1) Posts

To get the actual client IP, you need to pull it from the header.  Apache sets a header variable called “x-forwarded-for” with the client IP.  You can simply grab it from the header:

String ip = request.getHeader("x-forwarded-for");

This is just one of the thing I found annoying. I can understand why request.getRemoteAddr() doesn’t always return client IP. I’m hoping eventually Tomcat will have a reliable method to return a proper client IP.

Join the forum discussion on this post - (1) Posts

The most basic form of architecture for a Solr-based application only require a single application server. Assuming you develop in Java, you can have both Solr and your webapp served by the same application server. A more common and effective architecture would involve an dedicated indexing server (or indexer) and one or more slave index servers. The idea is to separate all index building work from normal queries. Conceptually, this is similar to database clustering where you have a read/write server as master and read-only servers as slaves.

The following set up involves Tomcat, Apache and Linux assuming Solr’s home is under /solr on every Solr servers.

Note: you may be able to replicate similar configuration on a Windows environment running Cygwin. I haven’t tried it on Windows yet so YMMV.

• Scripts configuration
  • Environment can be configured in solr/conf/scripts.conf. Here is a sample indexer configuration:

  user=solr
  solr_hostname=indexer
  solr_port=8080
  rsyncd_port=18080
  data_dir=data
  webapp_name=solr
  master_host=indexer
  master_data_dir=/solr/data
  master_status_dir=/solr/logs

  • Sample slave server configuration:

  user=solr
  solr_hostname=slave1
  solr_port=8080
  rsyncd_port=18080
  data_dir=data
  webapp_name=solr
  master_host=indexer
  master_data_dir=/solr/data
  master_status_dir=/solr/logs

• SSH set up
  • Solr uses SSH and Rsync in its index distrubtion scripts so we need to make sure SSH keys are configured and public keys are exchanged between indexer and slave index servers. If you haven’t configured SSH key yet, use the ssh-keygen command to generate public/private key pair on every Solr servers.

  $ ssh-keygen
  Generating public/private rsa key pair.
  Enter file in which to save the key (/home/solr/.ssh/id_rsa):
  Enter passphrase (empty for no passphrase):
  Enter same passphrase again:
  Your identification has been saved in /home/solr/.ssh/id_rsa.
  Your public key has been saved in /home/solr/.ssh/id_rsa.pub.
  The key fingerprint is:
  0c:27:27:f5:81:36:87:82:0f:4f:39:b5:aa:fd:e4:2f solr@solr

  • Exchange public key between indexer and slave index servers

  $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  $ chmod 644 ~/.ssh/authorized_keys
  $ ssh solr@indexer "cat .ssh/id_rsa.pub" >> ~/.ssh/authorized_keys

• Rsyncd set up
  • Solr uses rsync for index distribution so you need to make sure rsync is functional in your operating system. Start Rsyncd the first time with following commands:

  $ /solr/bin/rsyncd-enable
  $ /solr/bin/rsyncd-start

• Configure Solr to automatically generate a snapshot after optimize. Update solr/conf/solrconfig.xml with following:

      /solr/bin/snapshooter
      /solr/bin/
      true

• Enable snapshot pulling on slave servers:

$ /usr/bin/snappuller-enable

• Set up snapshot pulling on slave servers at 3am in cron:

0 3 * * * /solr/bin/snappuller; /solr/bin/snapinstaller; /solr/bin/snapcleaner -N 3

• OPTIONAL: set up Apache load balancing of your slave index servers (running Tomcat), update /etc/conf/httpd.conf with following:

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
....

    ProxyRequests Off
    ProxyPreserveHost On
    ProxyPass / balancer://tomcats/ stickysession=JSESSIONID lbmethod=byrequests
    ProxyPassReverse /  balancer://tomcats/
    
        BalancerMember ajp://slave1:8080 route=jvm1 loadfactor=20
        BalancerMember ajp://slave2:8080 route=jvm2 loadfactor=20
    

All indexing work should be done on your indexer. When you issue the optimize command, Solr will automatically generate a snapshot. Snapshot should be generate well ahead of the scheduled snapshot pulling time (3am in this case). Apache load balancing is optional if you only have one slave server or you have other load balancing solution.

Reference links:

http://wiki.apache.org/solr/CollectionDistribution

http://wiki.apache.org/solr/SolrOperationsTools

Join the forum discussion on this post - (12) Posts