A large part of WorldEdit uses an abstract internal API for easy portability  For your convenience, we provide a Bukkit-native selection API that makes it easier to work with that part of WorldEdit. It is possible to “reach through” to access to the internal features of WorldEdit, but if you are interested in only getting selections, that shouldn’t be needed.

Getting a selection

To get a selection, you need to:

1. Get a reference to WorldEditPlugin from Bukkit’s plugin manager.
2. Cast the reference from JavaPlugin to a more specific WorldEditPlugin.
3. Get the selection by calling getSelection().

See the snippet below.

Remember that the user may not have a selection yet, in which case, you need to detect it and notify the user.

What kind of shape?

Selections may be cuboids — that is, they may be boxes aligned to the world. However, they may also be 2D polygons extruded vertically. Depending on your use case, you may not care (you can just get the min/max points), but if you do, the selection object returned is actually one of a more specific kind. By using “instanceof,” you can detect which kind of shape it is and act accordingly.

Be aware that new shapes may be added in the future and you need to be aware of changes.

Proper dependencies

If you are using WorldEdit with your plugin, you may need to require that WorldEdit be loaded before your plugin. That way, you can guarantee that it’s available when your plugin is loaded. There are two kind of dependencies supported by Bukkit: a hard dependency, that requires WorldEdit to be installed for your plugin to load, or a soft one, where the loader will try its best to load your plugin at the end, but will continue if the soft dependencies are not found.

The following would go into your plugin.yml for a hard dependency.

In the case of soft dependencies, you can specify it as so:

Be aware that the call to getPlugin() in the first snippet may return null if you are using a soft dependency.

API stability

We don’t expect to change any of this any time soon, which means you won’t have to figure out why your plugin broke because of WorldEdit! It won’t break when MC versions change either.

In our case, some people experienced network jitter, while others didn’t. Because it really depended on which hops the connections passed through, the idea was to force a connection through a better performing hop.

How do you do that exactly? By setting up a “proxy” Minecraft server at a strategic location, users who had network issues could connect to that instead, and that proxy server would theoretically have a better route to the real server. tl, dr; You can make your server perform better for everyone else in the world, not necessarily by leaps and bounds, but at least by a little!

This provided a sometimes massive decrease in both network jitter and sometimes latency issues. In our case, because the server was in Germany, Americans sometimes benefited a fair amount. This was especially true for those in Hawaii, which is quite a bit far from Germany, and a few Hawaiians noted that it was much better with the proxy.

How do you do it?

Originally, I modified the “signpost.py” Minecraft server I wrote (for emitting server-is-down messages) to pass connections through. We used to run multiple servers on different IP addresses linked together, and because Minecraft had started passing the hostname of the server that you are connecting to in the login packet, I could choose which server to connect to right when you connected to the proxy, saving us the cost of buying IPs. It was pretty cool!

Nowadays, I actually use fproxy, which touts itself as fast “generic tcp, load-balacing, IPv6-ready (reverse) proxy.” It’s small and works well, just the right size to toss on a dirt cheap VPS instance. For Linux users, you just need download the latest version, run “make” (make sure to install gcc and other build essentials), and use something like:

./fproxy 0.0.0.0:25565 -- 1.2.3.4:25565

That’ll redirect everything on port 25565 to the IP address 1.2.3.4.

One thing I need to note: you need to turn off the connection throttle on the server, otherwise the server thinks you’re spamming it (because all requests come from one IP). You used to have to do this by modifying a few lines in Minecraft’s server code, but Bukkit users can now adjust this in bukkit.yml by setting connection-throttle to 0. For Forge servers, I’m not sure if you can do it without modifying the code, as the case with vanilla. I actually have always removed the connection throttle whenever possible because it serves no real purpose but to confuse users who are rarely aware of such a throttle.

If you run multiple servers, you can setup SRV records and setup several proxies.

Anything else I can do?

Yes! There are actually some other things you can do, but they require a bit more of an introduction. I’ll have to write it in a later post when I get some time.

As some who I have helped may know, aside from working on WorldEdit and friends, I’ve been a pretty big performance nut, having been the first person to write a /tps command for Minecraft! Reading through this article, you’ll find:

1. An overview of the game logic loop
2. How this all applies to modded servers
3. An explanation of transient problems
4. A guide to profiling your server
5. A macro-view of improving performance

The game logic loop

The most important part of Minecraft is the game loop. In there is a clock keeping everything synchronized, going tick-tock at an ideally constant rate: twenty times per second, with each tick or tock lasting hopefully at most 50 milliseconds. During each cycle, game elements have to do their work — skeletons have to figure out where to walk, minecarts have to travel forward, grass has to spread, and light problems have to be checked. That’s a lengthy to-do list, considering how many blocks and entities are on a loaded world at at time (thousands). This tick-tock interval is called the tick rate or “TPS,” and you can get some mod or plugin to give you your server’s rate.

Quite a few things are on the to-do list, but as far as the “main game elements” go, those items are blocks, entities, and tile entities. Blocks include simple things such as sandstone, cobble, sugar cane, and things of that nature. Entities are your standard animals and monsters, in addition to paintings, item drops, and “objects that don’t fit perfectly on a grid.” Tile entities are a special kind of block, as they can hold any type of data (think storing an inventory, something sugar cane doesn’t have to do), and sometimes they must think separately. As previously mentioned, each loop has a budget of 50ms and each portion consumes some part of this pie. Any leftover time is considered idle.

Usually on an un-loaded server, it works out and you have a nice slice of idle time, but what happens when it doesn’t “work out?” That “idle” slice in the picture above is nowhere to be found — it’s simply skipped over. That means that there’s no unused time anymore, and each clock cycle consumes more than 50ms. Things just slow down because they happen less frequently (including the passing of time!). While bits of Minecraft are multi-threaded (chat, for example), the game logic loop, the part you really care about, happens in one thread only.

Fundamentally, the problem is that of “too many items” with “each item taking 10 nanoseconds too long.” You have two approaches:

1. Reduce the number of entities and blocks loaded at any time.
2. Optimize the code for each block or entity to do more in less time.

Reducing numbers

One very important thing to note is that the number of loaded chunks correlates strongly with the number of blocks and entities. A reduction in the number of loaded chunks is a reduction in the number of offending objects!

How do you go about reducing numbers?

1. Unload chunks more often and more reliably. Surprisingly, doing this makes a world of a difference because Minecraft tends to “leak” chunks. This means that over time, chunks continue to stay loaded, adding more entities and blocks. It’s an extremely old bug that’s been known around for forever. Spigot is one third-party server (based on Bukkit) that provides an option to do this, though several Bukkit plugins have provided it in the past. You can also restart the server occasionally to “fix” this problem.
2. Reduce the chunk load radius. While this should be a last resort option, the view-distance option in server.properties adjusts the chunk load distance, which exponentially decreases the number of chunks to load every time you reduce the value by 1. This can be extremely effective.
3. Design your gameplay around reducing numbers. Your choices determine what your players will do, which may include building massive sugar cane farms that create hundreds of sugar cane blocks that drop thousands of sugar cane items.

Contrary to a popular belief, the number of players on a server does not correlate that strongly with “lag.” Players themselves do not use that much CPU, and players in the same area share the same loaded chunks. The number of chunks is what you should concern yourself with.

Just a note: While not many people can do this, turning off monsters (and animals) basically erases a big part of the pie, which has a massive benefit but the high cost of losing those lovely creepers.

Optimizing the game

Okay, so you’ve gotten the number of sugar canes down, but each sugar cane is still taking a little too much! Each one might take only 0.01ms, but when you still have hundreds of them, they add up.

The process of optimizing the entities and block themselves is not that easy; we’re talking about reducing entity or block “think time” by mere nano-seconds. However, there are people who work on the problem, to which you can benefit from their progress. For example, the Spigot project has a few of these tweaks, and if you run a vanilla server, those changes can make a bit of a dent in reducing lag on your server.

Other portions of the loop

While I’ve mentioned the blocks and entities, there are other things that go on during the loop that you need to be mindful of, and a few things you may want to do are:

1. Possibly consider disabling random “re-lighting.” As some may remember, Minecraft used to have numerous lighting bugs (dark spots) all over the place after its last major lighting update (barring the newest 1.5 update). The “fix” was to re-check lighting constantly across all chunks. Disabling this (with Spigot or by commenting out a few lines in the logic loop) can have a massive benefit with an acceptable level of lighting bugs. I can say that I’ve been doing this since this update came out (the impact was quite noticeable) and no one has really complained. (Note: Minecraft 1.5 has a new lighting system and this point may be obsolete).
2. Look into multi-threading chunk loading. Loading chunks requires them to be read from disk (a potentially very slow operation for magnetic disks), uncompressed, and then re-constructed into chunk, entity, etc. objects. Remember that chunks are constantly being loaded (and saved), because players are always moving even slightly across chunk borders. If you are using Bukkit, you already have the benefit of multi-threaded chunk loading. Otherwise, the best vanilla and Forge server users can do is to at least put the server’s world files on an SSD or RAM disk, and deal with the performance hit of decompression and object initialization.
3. Be mindful of chunk generation. Chunks have to be generated as the world is explored, which can, depending on how many people there are, eat quite a bit of CPU. The older your world is, the less of a problem this is, but if you are starting a new one, you may want to pre-generate chunks, and for all other cases, you may want to set a maximum world border.
4. Take note of the plugins that you use. Sometimes they just have to do what they need to do too (i.e. a skeleton can’t think in “zero” time either) to keep synchronized with the game, and you have to be aware of the costs involved.

There are other things going on in the loop, but they are not as worth mentioning. I may visit them in the future.

Now, modded servers?

If you do not run a modded server, you can skip to the next section.

If running a vanilla Minecraft server was not enough of a headache, it doesn’t get any better with a modded server (using mods from FTB or Tekkit, for example). Aside from the incredible number of exploits that come and go, the sometimes endless number of world crashing bugs, the unhelpful personalities that you sometimes have to deal with, getting the server to run smoothly can give you a few bald spots. But hey, it’s more fun.

With mods, the main change is that you know have a lot more entities and/or tile entities. (Though if you use the Forge server, you may also lose out on the improvements in Bukkit’s ecosystem, unless you use MCPC-Plus.)

Here’s the problem though: it’s a lot more. You still want to reduce the number of items and optimize each one, but it’s much difficult when you have hundreds of new blocks and entities to manage (plus you often don’t readily have access to the code of mods). Rather, you have to take a differing approach when dealing with modded additions:

1. Understand what the mods you install do, choosing wisely. Some mods, simply because they do more things, require more CPU. A flower mod? Probably doesn’t use that much CPU. A mod that simulates a tornado? Possibly a problem.
2. Design gameplay around the more CPU-expensive features. Sometimes you still want the mod, even if it’s expensive. The best course of action here is to design your server around them to discourage people from using more CPU-expensive features. You can also downright disable features, but that makes for sad players.

What makes a mod expensive?

Many mods have the potential to be both extremely cheap and expensive, performance wise, because it all depends on how the mod is utilized. To begin considering the potential cost of using a mod,

1. Consider what the additions in a mod actually do, and what “algorithm” might have to run on every tick. For example, an item that has to travel through a pipe has to find destination pipes and then inch forward every tick. That’s not actually that bad, but contrast this with liquid pipes, which may have to “balance” liquid throughout the pipes, although liquid pipes are still simpler than many other things.
2. Profile the Minecraft server (and see function call elapsed times) so you know what actually does take up the most CPU, either confirming or disproving your theories. We’ll get into this later.

Here are some examples:

• Logistics Pipes is a wonderful mod that automatically can route items between machines and inventories to do useful things such as keeping machines stocked with fuel, or distributing items across a network of chests. This, however, means that each item that enters the system requires that a destination for it be found among possibly a spider web of pipes. Supplier pipes, which routinely request from the network an item to stock a chest or machine with, can eat a lot of CPU time.
• RedPower2 allows the creation of “frame machines” that can move many blocks at one time. Combine this with its block breakers to make an automatic mining machine. Besides the obvious problems this involves, imagine the load demanded by a machine that is moving across the world and mining large numbers of blocks (which themselves trigger physics events).
• Greg’s Floodlights allow the lighting of long tunnels up to the light level of skylight. As Minecraft’s lighting system is fairly simple, the way the floodlights accomplish this is by placing invisible light blocks along the length of the beam. Lighting, for one, is expensive in Minecraft, and to add to that, the flood lights can be triggered by redstone.

That’s not to say that there is anything wrong with those mods. You probably might want to install them! They are fun. Instead of removing them outright, design your server’s gameplay around the more problematic features so as to discourage people from using them. For example, some possibilities would be:

1. Making Buildcraft quarries cheaper to entice people to prefer them over RedPower2 frame borers (although this is not as effective as you’d like).
2. Providing alternative ways to make cobblestone for IndustrialCraft2 scrap generation that do not require RedPower2 block breakers (which can get very expensive when you make a 100-breaker array!). Thermal Expansion provides a lag-free method of generating cobble, for example.

Transient problems

We’ve so far discussed problems that are always ongoing throughout the lifetime of a server, but some issues come and go. There’s no point trying to tackle the ongoing problems until you have a handle on issues that may be pretty easy to solve. Some examples include:

1. Item drops; They too are entities, and they consume CPU time like any other entity. With thousands of item drops on the map (which may occur due to massive sugar cane farms, for example), this can have a very significant impact on the server. Those who run modded servers are very familiar with this problem.
2. Massive numbers of creatures; Massive mob grinders or animal farms can collect hundreds of entities that all eat up some slice of the CPU’s pie. You may need to occasionally prune monsters that are bunched together, or otherwise design gameplay mechanics accordingly to discourage people from making these farms.
3. TNT; Massive TNT explosions may or may not have an impact on your server, depending on the server and the set of plugins that you use. A lot of servers outright disable TNT too. If you happen to allow TNT, you might want to conduct some tests.
4. RedPower2 tubes; (modded servers only) RedPower2 tubes never drop items, which is great, but sometimes the items get stuck bouncing around the pipes. This is not that easy to do, but once in a while, someone does it on accident and it will completely murder your server (and also consume a ton of network bandwidth).

That’s merely a sampling of the number of transient problems that you may have. Usually one of the best ways to identify transient problems is to profile your server, which we will cover next.

Be aware that WorldGuard does have a feature, /stoplag, that is basically the heavy-handed solution to a lot of vanilla transient issues. It will wipe all dropped items and monsters continually (even in newly loaded chunks), until you turn it off with /stoplag -c. (Warning: It will wipe all animals, as well as NPCs if you use Noppe’s NPC mod.)

Profiling Minecraft

Wouldn’t it be great to see actual numbers?

One plus with Java is its large ecosystem of tools for inspecting a running Java application. JVisualVM is one popular utility, and its “sampling” feature is extremely beneficial to identifying exact sources of server lag. What it does is simply to very periodically ask a Java process “what are you doing?” and then record the results, which would show, for example, if skeletons, were using 5% of the CPU time. Sampling is not accurate as “profiling” (which instead records every call as opposed to asking periodically), but it’s more than sufficient in this case (and a whole less server killing).

Note: This guide assumes that you are running JVisualVM on the same system as the server, or at least have access to a desktop environment, but this is not an option for many people. In my case, I have a very light VNC setup on the server to access JVisualVM directly, but you can try passing JVisualVM over SSH (which doesn’t seem to work very well for me). Another option is to use X11 over SSH, but I find this horrifically slow. There’s also the option to use FreeNX. If you are using a game hosting provider, you are completely out of luck.

You can get JVisualVM by downloading the Java Development Kit. Once you install it, you will find “jvisualvm” or “jvisualvm.exe” (for Windows users, look in C:\Program Files\Java\jdk#.##\bin, or in “Program Files (x86)” for 32-bit).

You’ll notice on the left side a list of currently open Java applications, and you should find your server somewhere on there. Double click the process.

Open the “Sampler” tab and click the “CPU” button to start collecting data.

You will want to wait for as long as you are willing to wait to get more accurate data, and then click “Snapshot.”

What you are interested in is Server thread. It is where the game logic loop resides. Expanding that node should give you the first function called (remember, in a program, functions call functions, which calls functions, which calls more). You will want to expand it until you get to the meaty part (but stick to expanding the nodes that are consuming the most CPU time).

It’s noteworthy to see that sleep[native]() is consuming 95.5% of the CPU time in the above screenshot. This is good — it is that “idle” part of the game logic loop, because I generated this screenshot on a server with nothing on it. You’d be in a great shape if your sleep() call showed up on the results with such a high percentage!

Why are the names strange? What does “r()” do? “b()”? Be aware that Mojang obfuscates the names of the code in the game (presumably to slightly complicate code theft), you will have to deal with this problem. Once you start profiling your server more often, you will learn to know what goes where. Please note that I ran this on a Bukkit server, and CraftBukkit de-obfuscates the class names (so you see MinecraftServer of MinecraftServer.r()), but you would see something like ayy.r() on a vanilla or Forge server. (If you’re on a Forge server or vanilla server, you can look at the files in conf/ from MCP to translate the names.)

Those are screenshots of an empty server: what would a real server look like? Here’s one on MCPC-Plus (which is a combination of Bukkit and Forge), and you can see easily what is consuming the most CPU time. It’s a modded server, and so you will see modded additions in the screenshot below:

I’ve expanded it to the level where entities and tile entities are processed. What we can tell from this screenshot:

• Buildcraft (which is a mod) is eating 21.9% of a tick’s time on its pipes.
  • However, most of that time is actually spent on the Logistic Pipes mod.
• 11.6% of the time (md.j_) is spent on living entities (skeletones, sheep, etc.). “md” corresponds to the living entity class. I used MCP to translate the name, as MCPC-Plus uses obfuscated class names.
• 12.7% is spent on RedPower2 motors. This corresponds to mostly RP2 frame boring machines, which is consuming about 12% of the entire server’s CPU time (in the main thread),
• Even though entities take up 17.5% (yc.g()), much of that percentage under that is taken up by items that take less than 1% of the time. That’s the problem of just “too many items.”
• You could go on and analyze more of what is going on, but for now, that should serve as an example.

Powerful information, isn’t it?

Can I blame java?

No, sadly not. While Java (or rather, the Java Virtual Machine) is known to use more memory than it may need to, CPU performance wise, it is very comparable to native code that is generated by C++ or C. That means that a port of Minecraft would not do you any good! (A rewrite would, but so would a rewrite of MC in Java.)

One thing I have neglected, however, to mention is that Java may need to run the “garbage collector” (that identifies unused objects and frees up memory) and it would tie up the game logic loop too, although it is not part of the loop. Before your eyes light up any further through, the JVM is defaulting to the multi-threaded garbage collectors now, meaning any lag that you may be experiencing is likely caused by other issues (unless you are on a single-core machine). That said, you can tweak the garbage collector in use, or even explicitly set one.

There are many articles and resources on how to tune the Java GC, but a lot of the options are more suited to optimizing long-running “enterprise” Java applications with uptimes of 10 months. That is not to say that you won’t benefit, and it’s something worth to try. Off the bat, I only suggest at minimum two flags (which, as I said, may be redundant):

java -server -XX:+UseConcMarkSweepGC -jar minecraft_server.jar

You’ll find a lot of the Minecraft-related threads on the Internet about what flags you can use, and even I wrote one a long, long time ago, but make sure that you understand what the flags you are using do, as sometimes people even suggest putting conflicting flags together!

Note: Java 7 has a new garbage collector named G1. I’ve heard good things about it, from people who run enterprise applications, and terrible things about it, from people who use it on Minecraft. I’ve had a poor experience using it on Minecraft myself. The flag I presented above does not use G1, but instead chooses the older Concurrent Mark and Sweep GC.

The big picture

We’ve been tackling the problem of performance on a micro level — looking into internals and what not — but let’s step back and see what else you can do.

1. Throw more money at the hardware.
2. Configure your operating system accordingly to reduce overhead.
3. Multi-thread the game logic loop so it can be split among several processors or cores.
4. Run several servers and link them together.

Hardware counts, somewhat

There is a limit to how much money you can throw at the hardware, at least until Minecraft’s game loop is multi-threaded. Your problem is that you do not need more cores — you need cores with faster clock speeds (think 10 Ghz), but you’re not going to find those kind of processors to buy, and at best, you can try overclocking (for home-hosted or colocated servers). If you have the funds, buy the fastest processor that you can get your hands on (preferring clock speed over the number of cores).

Aside from the processor, RAM does matter, but it’s more of a “minimum” that you need. If you do not have enough RAM and the server has to be swapped to disk (put into virtual memory), the performance hit is massive because the disk may be too slow, and you’d be in contention with the chunk loading. Once you have enough RAM, throwing more RAM at the server won’t do much at all.

However, hard disk speed, especially if you do not have multi-threaded chunk loading, can have a very big impact. This is actually where a lot of virtual private servers (VPS’s) and game server providers (GSPs) fail to meet requirements, because a lot of chunk loading and saving is being blocked by waiting on the disk (which also consumes CPU time, counted as “I/O wait”). If you’re intending on purchasing a new VPS (or even a dedicated server) for Minecraft, you should test its read/write speed (and several times over a period to find an average).

You can get around disk speed problems by using a RAM disk, which essentially makes a part of your RAM into virtual disk space. RAM is absurdly fast, but it comes with its own problem: everything in RAM is lost on server reboot, and so you need some way to frequently copy the server back to disk. If you use a RAM disk, make sure to not underestimate the amount of space that your world will need too. You can use an SSD instead, which is nowhere as fast, but still considerably quicker than magnetic disk (but be aware, SSDs don’t like a lot of writes, which Minecraft is a prime user of).

Reducing system overhead

Even with the right hardware, you may want to tweak your system to reduce a bit of overhead. These tips are Linux-oriented, but I’m sure users of other operating systems may have some similar (or related) courses of action:

1. Consider enabling Linux’s “hugepages.” It reduces the overhead of accessing memory by increasing the size of Linux’s fundamental memory block size. Check your search engine on how you can do this on your particular distribution.
   
2. Reduce Linux’s swappiness value. Even if you have a massive amount of RAM available, you may notice that sometimes Linux still swaps programs to disk.  If you have plenty of RAM to blow (with usually a safe amount of free RAM, to prevent sudden excruciating swaps), you can try setting the swappiness factor to 0.
3. Tune tcp_wmem/tcp_rmem accordingly. This was suggested to me (by /u/chiisana), and I’m afraid that I’ve never considered this! See this article for more information. It might help a bit overall.

Multi-thread all the things

You can multi-thread the game logic loop, but this is not an easy task (and worth hundreds of hours of work). The reason is that making algorithms parallel is quite hard work, as seen below just making a simple counter parallel:

Now apply that to hundreds of game blocks and entities, especially if you have mods. None of them expect it, and if you were to put them into different threads, they would all step on each others’ toes and inevitably corrupt something. Some have nevertheless tried (see nallar’s TickThreading project), and a few have made progress. Unfortunately, it is largely a hack until the entire game is redesigned from the group up with multi-threading in mind. (You can try TickThreading if you wish though, but sadly it did not work for me on a large pre-existing world.)

More servers

Lastly, another solution is to run several servers and link them together. Many people have done this in the past, by making portal mods, shared chat mods (something I used to do), and linking serves through one common proxy server (see BungeeCord). At some point, this is your only option, and so you may want to consider organizing your server to be cash flow positive in order to pay for more servers.

In conclusion

As a summary of this post,

1. Make sure you are running on good hardware (esp. for shared systems).
   1. Ensure you have a dual-core (or processor) or better CPU available, with the fastest (and reliable) clock speed that you can get your hands on.
   2. Make sure you have enough RAM so that Minecraft isn’t swapped to disk.
   3. Make sure that your hard disk throughput is high enough, or consider using a RAM disk.
   4. Consider tweaking your operating system to reduce overhead.
2. Make sure to identify and remove transient problems. If your server starts lagging, look for the transient problems first and foremost.
   1. Are people building things that drop too many items?
   2. Is there some giant collection of 300 sheep somewhere on the server?
   3. Did someone sent off a ton of TNT?
   4. Is there some massive flow of water or lava somewhere?
   5. For a modded server, are people building poorly-designed pipe systems? Frame borers?
3. The next step is to tackle ongoing problems, and to repeat the bolded entries from above:
   1. Install Spigot if possible.
   2. Add in an auto-restarter if you do not have something to unload chunks automatically.
   3. Reduce the view-distance setting in server.properties.
   4. Design your gameplay to incentivize people to build less demanding things.
   5. Disable random “re-lighting” of chunks if possible (with Spigot, possibly).
   6. Use Bukkit if you aren’t already for multi-threaded chunk loading
   7. Pre-generate chunks if necessary, and/or set a world border.
   8. Go through mods and plugins to see if one of them is spending more time than you can handle at the moment. Contact their developers and see what options they can provide you (and be nice! writing fast code is not as easy as it might sound sometimes.)
4. When in doubt, use JVisualVM (or another profiler, of several available, but JVisualVM is free) to find out the exact cause. In fact, you should use a profiler whenever possible.
5. If you just can’t squeeze enough performance out, consider making several servers.

(On a relevant note, if you play with mods and your frame rate is really low, you can try using JVisualVM on the client too to figure out which mod is using the most CPU and eating into rendering!)

It’s all web-based and powered by HTML, and so with a few changes to the page to make it more compact…

To do the same, you’ll need Stylebot if you’re on Chrome, or something equivalent for other browsers (Stylish for Firefox). After installing Stylebot, navigate to the Steam chat page, click the “css” button in the address bar (it’s to the right), click the “Edit CSS” button on the new sidebar, and then paste in the code below:

If you are using the application shortcut, the CSS changes made in the main Chrome browser will also be reflected there.

We briefly tested out Assembla 2-3 years ago to replace the tracker. It is more of an overall solution than simply a bug tracker, providing a “workspace” with optional support for its own hosted repositories, wiki, messaging, and file uploading. The features were optional and we were able to disable what we did not need. Unfortunately, the interface was cluttered – we felt that the interface was built more for the developer than your public issue reporter. As the goal was to promote the reporting of bugs and feature requests, we were hesitant to continue using Assembla. One proposal was to make a more user-friendly frontend, but we felt that it was outside the scope of our responsibility. In addition, my main concern with ditching GitHub was that it meant that new issues added on Assembla wouldn’t appear to users on GitHub who had starred the project (the community aspect is useful for an open source project), meaning that we weren’t far from the tipping point for the decision. It was not too long until we decided to return back to GitHub.

Sometime later, once our issues came to light again, we decided to try out Redmine. A popular open source Ruby-powered issue tracker, it had a few things going for it:

• There was a copy I had installed around in 2007 that seemed to still work. As marginal as that sounds, installing Redmine has never been a particularly welcoming experience, requiring that you not mismatch a minor dependency version without completely breaking the installation. (Redmine has broken twice in the past for us due to some not-entirely-relevant updates.)
• It was free to us and open source. Granted, as a not-for-profit open source project, we had many options available for us and this was not unique to Redmine, but it was definitely a plus.
• It was easily themable if we wanted, and it was decently pretty out of the box. While function is important, aesthetics and usability was important to consider as well. Redmine seemed to fit the bill.
• The interface of Redmine has historically been pretty straightforward. There’s not too much on the screen: just the issues, and registering and posting a new issue seemed easy enough.
• We could define fields however we wanted, forcing people to make the right choices when prompted with the New Issue screen. Every one less mistake users made in their report was one less mistake a developer or issue tracker moderator had to fix.

All seemed good, except for a major inconvenience: having to manage hundreds of issues quickly with Redmine was a chore. It supported Ajax requests – right clicking issues and choosing new states – but it required a new page refresh. If I had to make multiple changes to many requests quickly, it meant that I had to rely on the old “shift-middle-click” trick to open each change request in a new browser tab, sometimes requiring 40 tabs open in the process. Searching for issues was not the easiest and we lost the ability to tag issues entirely by switching over from GitHub. Everything just felt more of a struggle.

For all the features it had over GitHub, it lacked some more advanced features we desired, such as a proper issue workflow configuration.  Issues shouldn’t jump between different states randomly, especially as initiated by a user. We had users marking the priorities of their own issues as “CRITICAL” when it was anything but. The problem was no longer so much as users not putting enough tags, but now it was that users were incorrectly setting the properties of their issues.  A number of these bugs or features affecting us had been reported on Redmine’s own tracker, but many of those issues were untouched for years (possibly because their own developers found managing the issues difficult!, or so I can conjecture). Redmine was anything but a dead project, but it was not moving quickly enough for us.

Nevertheless, we trudged on with Redmine for over a year. We always wanted to try something new, but we weren’t willing to disappoint ourselves by choosing the wrong one. Bugzilla seemed to confuse the heck of the non-savvy folk, FogzBugz wasn’t free for open source projects, Trac wasn’t an option for a few reasons (which I don’t recall), JIRA bothered us with a number of smaller issues, Mantis Bugs was just ugly (sorry), all the other smaller SaaS issue trackers seem too young and lacked far too many features (which was sometimes billed as “less clutter”), and the list just goes on.

Enough was enough

Towards the end of last year, I had taken a short break prior and returned to the tracker, finding it filled with hundreds of simply invalid bugs that everyone else had appeared to stop caring to prune. I began the task of pruning myself, but at some point I decided that it was just not worth it. We needed a new issue tracker.

Years of dealing with the problem meant that we had weeded out most of the other issue trackers and we ended up choosing between three:

• Bugzilla, because it has a long history and was used by a number of large organizations. It also had improved a lot over the years, even taking on a new default theme so that it would no longer look “from 1995” out of the box.
• JIRA, simply because it had a very large feature set and filled nearly all our needs. It had a very capable workflow editor and provided support for numerous 3^rd-party plugins.
• YouTrack was actually new to us. None of us were aware of it previously, but perhaps because the tracker had made huge gains in features in the previous few years. We were familiar with JetBrain’s other work, and we were generally impressed.

I installed the trial versions of JIRA and YouTrack and asked various people to play with it and provide me with their opinion. Bugzilla was already familiar to everyone and so I skipped over a trial installation for that (and because I didn’t want to set up a Perl script again). The general response was:

• People either loved or hated Bugzilla. This wasn’t really useful to making a decision, and so we just put Bugzilla at the bottom of the list for the time being.
• The most common complaint about JIRA was that it was “slow.” On load, it definitely required a lot of resources, but once everything had been cached, the interface was fairly quick. I’m sure on a local network, there would not be an issue at all, but this would be hosted over the Internet and perception was everything. Because of that, there was quite a bit of resistance to JIRA.
• People commonly complained about YouTrack’s interface being “too different.” It doesn’t normally use a table-based view, something many people are used to, and the new issue screen has a radical display where the field names (type, priority, milestone) are not displayed, which worried us as to whether people would start plugging in the wrong choices. It was just all around different.

Choosing the right one

YouTrack was, as you may know, our eventual choice, and we don’t regret it. Everyone got used to the interface and it was not a problem at all. Users didn’t have a problem with navigating the tracker, as it was so straight forward (and yet powerful). One cool thing about YouTrack is that it has a proper downloadable Workflow Editor utilizing its own powerful DSL, which was miles more convenient than having to work through JIRA’s numerous dialogs to setup a simple workflow. In fact, I had grand plans to set up an elaborate workflow to prevent users from misreporting issues, but we ended up only needed a few simple rules.

Why? Navigating issues was a breeze, and the problems we had with people misreporting bugs turned out to be not as big of a problem anymore. The best part is probably the search bar at the top: you can filter issues exactly how you want using its expressive search language. Also partially command driven, rather than having to click through several dialogs to complete more complex tasks, you can just type a few commands in the right place. Before every release, we make sure to look through certain searches to make sure that we have not forgotten anything, something we didn’t find as easy to do with other trackers.

• Easily spread around the work of updating the server (or corresponding client) with others. No more figuring out how to merge changes when you need to send files to other team members. Compared to Dropbox, you have far more tools available to you with a proper version control system — easily see differences down to individual lines.
• Track every “release” of your server, which means that you can know exactly what was updated on what day, and most importantly, allow you to rollback to a prior release without having to scrounge around for the old files or old configurations.
• If combined with a “deployment script,” you can just plug in a version that you want (i.e. release-2013-02-20) and your server could be automatically updated. Did the update go badly? Return to an old version (release-2013-02-16 perhaps?) instantly.

When we need to update the server (or client, in the case of a modded server), it’s just a matter of a few commands. It is almost a luxury to do updates (well, at least until you realize mod X eats all the bandwidth and plugin Y has a dupe exploit you missed).

Picking a version control system

The most remarkable difference between version control systems (VCS) is whether one is “distributed” or not.

• With a non-distributed VCS, a central server stores all of the history and each person with access has a copy of the latest version. Changes made must be immediately sent to the server when “committed” (at least historically; some systems have updated to be more competitive with distributed systems).
• A distributed VCS does not require a central server to store the entire repository. Each person has an entire copy with all of the history, and each person can make changes independently, and multiple commits can be collected together before they are pushed to a central server. Distributed VCS software still allows for collaboration, but the server hosting the repository is not particularly special: it merely has the copy that everyone else synchronizes with.

I personally strongly recommend Git, a popular distributed VCS used for WorldEdit, Bukkit, and many other projects. It has good support for non-text files too, which is perfect when you need to store binary code (.jars and such).

Now that I’ve chosen Git for you

You need to get Git. For Linux users, you can get it from your operating system’s package manager. Windows users should install mysysgit (I recommend selecting “Use from command line” during install), but they should also consider installing TortoiseGit too, which allows right clicking folders in Windows Explorer and using Git with an actual GUI. Mac OS X users can get an installer and can also get a GUI too.

Furthermore, if you are working with others or wish to later “deploy” the changes to a server from that server itself (if you have a VPS or dedicated server), you may want to host a version of your repositories somewhere. It provides a central place for several people to synchronize changes with. Many services provide public open source repositories for free, but you probably are considering a private option.

• GitHub is arguably the most popular Git service provider. They charge for the number of repositories that you have, but allow for an unlimited number of collaborators, with a micro plan starting at $7/mo (as of writing).
• BitBucket charges by the number of collaborators, but allows for an unlimited number of repositories. They also have a free plan as well, which you may want to start out with.
• You can also set up Git on your own server if you have one. Linux users can setup Gitolite.
• There are also many other services that provide Git hosting, which you can look for. Assembla is another one.

How Git authenticates you is not through a password: you need to generate an SSH key, which is kind of like a super long cryptographically-generated password (but a bit more than that). You can generate a key using ssh-key-gen on Mac OS X or Linux, or with PuTTYgen on Windows. Two keys will be generated: a public one to give to the host that you may have chosen (GitHub, BitBulcet, etc.), and a private one you need to keep secret and that you will use to authenticate yourself.

Using Git

One minor downside to Git is its higher learning curve, but the effort will pay off. Let’s first review some of the words that you will have to know to start, some of which are common to all version control systems.

• The log is your log of changes.
• A commit is a set of changes that you make together and then apply a description to (i.e. “Fixed X and Y”).
• A branch is a “fork” of your repository, which a separate list of changes that can diverge from the main branch.
• A merge is the merging of changes from a different branch (to the top of the current branch’s list of changes).
• A conflict is when automatic merging fails because two people modified the same file at the same place.
• Pushing is sending changes to a server.
• Pulling or fetching is receiving the latest changes from a server.
• A clone is the process of downloading the entire copy of a repository.

Sadly, a full-blown tutorial for using Git is a bit outside the scope of this article, but Git has a wonderfully large number of resources. You can start by using this interactive Git tutorial and continue on by reading this Git book. Linux (and Mac OS X) users can also try this step by step introduction. If none of that fancies you, you can try searching the web, or possibly asking a few of your programmer friends :)

Be aware that Git is predominately a command line program, but if you had installed a GUI, you can try using that. The processes and terminology carry over; you just know need to press the right buttons!

On the subject of a deployment script, it is much easier if you control the server that Minecraft is hosted on. That way, you can simply download the latest changes from Git and copy them over to where the server’s files are stored. Otherwise, you may have to use something to upload the files over FTP. Sadly I do not have an example deployment script to provide, as mine is fairly specific to my case, but all you need to do (which you can even put as lines in a .bat file) is download the latest changes using the right git commands (i.e. git pull, or git checkout release-XXXXX) and then copy files as needed (or compile plugins, or whatever).

Running out of disk space

Does running out of disk space cause data loss? Not necessarily. Picture a chunk file as the example below, with three different chunks stored in different “slots.” The three squares represent disk space already “acquired” on the disk as one single .mca file, with the white areas representing acquired (from the PoV of a hard drive) but unused (from the PoV of Minecraft) space.

Sometimes a chunk needs to be enlarged, but only some activities do that. If you are placing down “simple” blocks like sandstone or cobble, you’re just changing a few bits of data in the chunk, so the chunk does not need to be resized. However, this is different when you add new entities (mobs, such as skeletons, or item drops, etc.) or new tile entities (more complex blocks, like chests, furnaces, etc.) because the existing space isn’t enough to store all the new data (think inventories!). Those changes require more disk space to store the chunk.

But even then, even if the hard drive itself is out of space, sometimes it is OK. The middle chunk above still has “room” to expand. The white space available was already, as mentioned previously, acquired on the disk.

The problems appear when the slot that the chunk is already contained within becomes too small to fit the newest version of that chunk. What happens is that a new free area has to be found in the file that can fit the chunk, but if that cannot be found, the chunk is placed at the end of the file. Placing data at the end of the file means that the entire file itself has to be enlarged, therefore requiring more actual disk space usage. When the hard disk is out of space, the extra data has nowhere to go. Because the header of the .mca file indexes where a particular chunk is stored, the index for this chunk may now point to this section of the file that cannot actually be written, causing the chunk to simply “disappear” on next load.

As seemingly irrelevant this information is to the world outside Minecraft, this also explains the same problem that your computer has when saving something to disk. When you save a file, it has the same, sometimes difficult, task: find a place to store the bits. If you’ve ever heard about “fragmentation,” that is when, rather than simply moving the entire file to another place, the file is split across across several areas. Of course, while Minecraft can put more data at the end of the file, your hard drive has a fixed amount of total space. The chunk format used by Minecraft does not do fragmentation.

Several months ago, I updated my Minecraft launcher to support custom downloads and open sourced the entire launcher. While the update system worked pretty well, there was one caveat: it required that downloads be digitally signed, similar in principle to the browser verification of yourbank.com truly coming from yourbank.com. Setting up digital signatures is pretty straightforward, but it’s a straightforward in the same vein as client installation – it is easy if you are familiar with the tasks involved. For that reason, as of today, I have decided to disable digital signature verification for custom packages, which should make it vastly easier (vanilla downloads from Mojang are unaffected). I was trying hard to avoid this, but other, more elegant options require a currently unrealistic input of time and effort from me.

tl;dr: I have made a change to my Minecraft launcher which vastly makes the use of the custom update feature easier to setup, which allows you to setup a custom modified MC client for people. Use it to make playing with friends less hectic, or use it to distribute necessary files for a modded server. After following the steps described within this post, you should be able to distribute your own files and install them into a user’s game. Towards the end of the post will be a discussion of some non-technical issues involved with this. Update to the latest version of the launcher to use this newest change.

Building the package

You’ll need three files:

1. A .zip file containing the files to install. This archive should contain files that would be placed into the root Minecraft folder (aka. .minecraft).
2. A package XML file containing a list of files to install, with a link to the ZIP archive.
3. An update XML file containing a link to the package file and a version string.

The package XML file is the more complicated file, as it has to contain all the files that need to be installed even on a fresh installation of Minecraft. You’ll see in the example file below three <filegroup> sections, the first of which is your main concern, as it points to the ZIP archive mentioned above. The other file group sections merely point the launcher to where additional required Minecraft files can be acquired. As you may notice, you would have to change the directory URL for the ZIP file, the filename of the ZIP file, as well as the size in bytes so that the launcher can accurately draw the download progress bar.

The latter update XML file is considerably simple. The version string you choose can be anything, but it must stay the same until you issue an update. Upon the launcher finding a different version, it will download the referenced URL to update. The URL points to the current latest package version.

Once you’ve placed all three files on a web server somewhere, fire up the launcher, create a new configuration, choose a folder to store the files, and insert the URL to the update XML file.

Installation mechanisms

The next question arises is that of “how does the launcher actually install the file?” It simply extracts the files over the Minecraft directory and overwrites any files. As for updates, the launcher keeps track of the list of files from prior updates and removes any files that disappeared between updates. A player’s saves would not be removed because it would not be part of the download ZIP, but any configuration change made by the user on a file added by the update would be wiped on next update. Overall, this process allows for very simple installation at the cost of some flexibility.

Support for incremental updates is there, as the resources downloaded from Mojang use that, but it is a bit convoluted and meant to fit in with Mojang’s existing download scheme. I may detail how it works in a future post, but for now, sticking to full updates is easier to get working.

The other issues

What about minecraft.jar? You have bundle that with the .zip (to keep it simple for you), which is not entirely kosher, but remember that you can already download Minecraft (without logging in) from assets.minecraft.net. However, this decision is up to you.

The second issue is that you may distribute other people’s work, which will inevitably happen, and so I can only ask of you to be sure that to have permission to distribute the files. The terms are often mentioned on people’s posts, but send them a friendly message if they are not clear about it.

There are currently a few rough edges with the launcher. The fact that you have to fiddle with the configuration screen is not ideal, and the fact that you can’t just two click add a configuration is another dislike from me. Letting users add their own client customization is also not so easy, although it could argued that you would not want to make that easy. If you distribute your own copy of the launcher (and make appropriate modifications), these rough edges disappear, but this is obviously not the best option. I will see if I can find some time to sand out some of the left over difficulties. Remember that you can contribute to the project, as it is open source.

A “session stealer” exploit in Minecraft is now being used increasingly widely to gain operator privileges on a target server. Anyone with the ability (or the freely available tools) can impersonate another user, which comes in handy to gain elevated privileges on a given server. The underlying exploit is actually extremely old, and while a small number of people have long been aware of it, news broke to the greater community recently and now it is a real relevant concern. On the flip side, the bug will finally be fixed (after three years) with the release of Minecraft 1.3. Hooray public disclosure.

But I don’t want to wait for a fix

If you run a server, it’s understandable that you would want a fix now. You can ensure that you don’t expose your own account, but you also have to ensure that your own moderators do not either. That can be harder than it sounds, especially depending on who you are working with, and so some — even minor — protection to use would be nice. Unfortunately, it is not possible to fix the problem from just the server’s side, and so people have attempted to work around the problem by using a mix of some rather inconvenient solutions (mostly involving a /login command). Instead, I decided on a different novel way to protect the accounts of my moderators, described below.

1. It doesn’t require any client-side changes;
2. It is a one-time setup for everyone;
3. It completely fixes the exploit for your server;
4. And it is grossly simple to setup.

All it requires is that everyone that you want to protect to switch to connecting with a secret server address instead. This solution exploits the fact that, when you connect to a Minecraft server, the address used is sent to that server. If your friend has a special address to use, and someone foreign tries to login to the server without that special address, the server has a clear indication that your friend is not really your friend. Some examples of address that you can use include:

• noHeY.example.com
• zqTPa.example.com
• 5v9h0.example.com
• nmJka.example.com

When an attacker tries the exploit:

So, how do I actually use this?

Before you setup the Minecraft server, you have to first set up the secret domains. There are, fortunately or unfortunately, a dozen different ways to do this, all depending on your particular setup. While I can’t delve deep into this in this post, I can recommend that you setup a wildcard DNS entry if possible, so all subdomains (*.example.com) get sent to your server. That way, an attacker can’t so easily guess domains and test their guesses. Another option, however, is to use your computer’s “hosts file” (search this up) in order to manually map any domain, even one that you do not own, to point to a certain IP address. This latter option is the safest, but it is also does require a bit more work from everyone involved.

On the Minecraft server side of this fix, I have added a check feature into WorldGuard (Bukkit required) that lets you check the connecting address of certain players, which you can apply to the accounts of your moderators. Download the latest version, and then setup your plugins/WorldGuard/config.yml to look like:

Be aware that this is, however, not a proper fix. It will protect a server, but it will not protect an account. If a session gets stolen, the attacker would not be able to use it on your server, but the account can be used on any other server. There is nothing much a server owner can really do (besides more drastic measures) and we are all waiting on Mojang to properly fix this problem.

You can find compiled versions (and a link to the source) on the MC forums.