Enzotools Collective

EnzoTools Blog: What's Up With yt 3.0?

matthewturk@gmail.com (Matthew Turk) — Thu, 15 Nov 2012 21:05:33 +0000

This is a long blog post! The short of it is:

If you're using Enzo or FLASH, you can probably do most of what you want to do with 3.0. But there are probably bugs, and you can't volume render yet. But every bug or missing feature you find is a useful piece of information that can help speed up development.

If you're using RAMSES, 3.0 will be a vast improvement!

Boxlib codes will need a small amount of updating to get working in 3.0

SPH is coming!

As with all of yt, 3.0 is being developed completely in the open. We're testing using JIRA at http://yt-project.atlassian.net/ for tracking progress. The main development repository is at https://bitbucket.org/yt_analysis/yt-3.0 and discussions have been ongoing on the yt-dev mailing list. Down below you can find some contributor ideas and information!

Why 3.0?

The very first pieces of yt to be written are now a bit over six years old. When it started, it was a very simple Python wrapper around pyHDF, designed to make slices, then export those slices to ASCII where they were plotted by a plotting package called HippoDraw. It grew a bit to include projections and sphere selection over the course of a few months, and eventually became the community project it is today.

But, despite those initial steps being a relatively long time ago, there are still many vestiges in yt. For instance, the output of print_stats on an AMR hierarchy object is largely unchanged since that time.

Most importantly, however, is that yt needs to continue to adapt to best serve analysis and visualization needs in the community. To do that, yt 3.0 has been conceived as a project to rethink some of the basic assumptions and principles in yt. In doing so, we will be able to support new codes of different types, larger datasets, and most importantly enable us to grow the community of users and developers. In many ways, the developments in yt 3.0 will serve to clarify and simply the code base, but without sacrificing speed or memory. By changing the version number from 2.X to 3.0, we also send the signal that things may not work the same way -- and in fact, there may be API incompatibilities along the way. But they won't be changed without need, and we're trying to reduce disruption as much as possible.

yt 3.0 is designed to allow support for non-cartesian coordinates, non-grid data (SPH, unstructured mesh), and to remove many of the "Enzo-isms" that populate the code base. This brings with it a lot of work, but also a lot of opportunity.

If you have ideas, concerns or comments, email yt-dev!

What's Going In To 3.0?

We've slated a large number of items to be put into 3.0, as well as a large number of system rewrites. By approaching this piecemeal, we hope to address one feature or system at a time so that the code can remain in a usable state.

Geometry selection

In the 2.X series, all geometric selection (spheres, regions, disks) is conducted by looking first at grids, then points, and choosing which items go in. This also involves a large amount of numpy array concatenation, which isn't terribly good for memory.

The geometry selection routines have all been rewritten in Cython. Each geometric selection routine implements a selection method for grids and points. This allows non-grid based codes (such as particle-only codes) to use the same routines without a speed penalty. These routines all live inside yt/geometry/selection_routines.pyx, and adding a new routine is relatively straightforward.

The other main change with how geometry is handled is that data objects no longer know how the data is laid out on disk or in memory. In the past, data objects all had a _grids attribute. But, in 3.0, this can no longer be relied upon -- because we don't want all the data formats to have grids! Data is now laid out in format-neutral "chunks," which are designed to support selection based on spatial locality, IO convenience, or another arbitrary method. This allows the new GeometryHandler class to define how data should be read in off disk, and it reduces the burden on the data objects to understand how to best access data.

For instance, the GridGeometryHandler understands how to batch grid IO for best performance and how to feed that to the code-specific IO handler to request fields. This new method allows data objects to specifically request particular fields, understand which fields are being generated, and most importantly not need to know anything about how data is being read off disk.

It also allows dependencies for derived fields to be calculated before any IO is read off disk. Presently, if the field VelocityMagnitude is requested of a data object, the data object will read the three fields x-velocity, y-velocity and z-velocity (or their frontend-specific aliases -- see below for discussion of "Enzo-isms") independently. The new system allows these to be read in bulk, which cuts by a third the number of trips to the disk, and potentially reduces the cost of generating the field considerably.

Finally, it allows data objects to expose different chunking mechanisms, which simplifies parallelism and allows parallel analysis to respect a single, unified interface.

Geometry selection is probably the biggest change in 3.0, and the one that will enable yt to read particle codes in the same way it reads grid codes.

Removing Enzo-isms

yt was originally designed to read Enzo data. It wasn't until Jeff Oishi joined the project that we thought about expanding it beyond Enzo, to the code Orion, and at the time it was decided that we'd alias fields and parameters from Orion to the corresponding field names and parameters in Enzo. The Orion fields and parameters would still be available, but the canonical mechanism for referring to them from the perspective of derived fields would be the Enzo notation.

When we developed yt 2.0, we worked hard to remove many of the Enzo-isms from the parameter part of the system: instead of accessing items like pf["HubbleConstantNow"] (a clear Enzo-ism, with the problem that it's also not tab completable) we changed to accessing explicitly accessing pf.hubble_constant.

But the fields were still Enzo-isms: Density, Temperature, etc. For 3.0, we decided this will change. The standard for fields used in yt is still under discussion, but we are moving towards following PEP-8 like standards, with lowercase and underscores, and going with explicit field names over implicit field names. Enzo fields will be translated to this (but of course still accessible in the old way) and all derived fields will use this naming scheme.

Non-Cartesian Coordinates

From its inception, yt has only supported cartesian coordinates explicitly. There are surprisingly few places that this becomes directly important: the volume traversal, a few fields that describe field volumes, and the pixelizer routines.

Thanks to hard work by Anthony Scopatz and John ZuHone, we have now abstracted out most of these items. This work is still ongoing, but we have implemented a few of the basic items necessary to provide full support for cylindrical, polar and spherical coordinates. Below is a slice through a polar disk simulation, rendered with yt.

Unit Handling and Parameter Access

Units in yt have always been in cgs, but we would like to make it easier to convert fields and lengths. The first step in this direction is to use Casey Stark's project dimensionful ( http://caseywstark.com/blog/2012/code-release-dimensionful/ ). This project is ambitious and uses the package SymPy ( http://sympy.org ) for manipulating symbols and units, and it seems ideal for our use case. Fields will now carry with them units, and we will ensure that they are correctly propagated.

Related to this is how to access parameters. In the past, parameter files (pf) have been overloaded to provide dict-like access to parameters. This was degenerate with accessing units and conversion factors. In 3.0, you will need to explicitly access pf.parameters to access them.

Multi-Fluid and Multi-Particle Support

In yt 3.0, we want to be able to support simulations with separate populations of fluids and particles. As an example, in many cosmology simulations, both dark matter and stars are simulated. As it stands in yt 2.X, separating the two for analysis requires selecting the entire set of all particles and discarding those particles not part of the population of interest. Some simulation codes allow for subselecting particles in advance, but the means of addressing different particle types was never clear. For instance, it's not ideal to create new derived fields for each type of particle -- we want to re-use derived field definitions between particle types.

Some codes, such as Piernik (the code Kacper Kowalik, one of the yt developers, uses) also have support for multiple fluids. There's currently no clear way to address different types of fluid, and this suffers from the same issue the particles do.

In 3.0, fields are now specified by two characteristics, both of which have a default, which means you don't have to change anything if you don't have a multi-fluid or multi-particle simulation. But if you do, you can now access particles and fluids like this:

sp = pf.h.sphere("max", (10.0, 'kpc'))
total_star_mass = sp["Star", "ParticleMassMsun"].sum()

Furthermore, these field definitions can be accessed anywhere that allows a field definition:

sp = pf.h.sphere("max", (10.0, 'kpc'))
total_star_mass = sp.quantities["TotalQuantity"](("Star", "ParticleMassMsun"))

For codes that do allow easy subselection (like the sometime-in-the-future Enzo 3.0) this will also insert the selection of particle types directly in the IO frontend, preventing unnecessary reads or allocations of memory.

By using multiple fluids directly, we can define fields for angular momentum, mass and so on only once, but apply them to different fluids and particle types.

Supporting SPH and Octree Directly

One of the primary goals that this has all been designed around is supporting non-grid codes natively. This means reading Octree data directly, without the costly step of regridding it, as is done in 2.X. Octree data will be regarded as Octrees, rather than patches with cells in them. This can be seen in the RAMSES frontend and the yt/geometry/oct_container.pyx file, where the support for querying and manipulating Octrees can be found.

A similar approach is being taken with SPH data. However, as many of the core yt developers are not SPH simulators, we have enlisted people from the SPH community for help in this. We have implemented particle selection code (using Octrees for lookups) and are already able to perform limited quantitative analysis on those particles, but the next phase of using information about the spatial extent of particles is still to come. This is an exciting area, and one that requires careful thought and development.

How Far Along Is It?

Many of the items above are still in their infancy. However, several are already working. As it stands, RAMSES can be read and analyzed directly, but not volume rendered. The basics of reading SPH particles and quickly accessing them are done, but they are not yet able to be regarded as a fluid with spatial extent or visualized in a spatial manner. Geometry selection is largely done with the exception of boolean objects and covering grids. Units are still in their infancy, but the removal of Enzo-isms has begun. Finally, non-cartesian coordinates are somewhat but not completely functional; FLASH cylindrical datasets should be available, but they require some work to properly analyze still.

Why Would I Want To Use It?

The best part of many of these changes is that they're under the hood. But they also provide for cleaner scripts and a reduction in the effort to get started. And many of these improvements carry with them substantial speedups.

For example, reading a large data region off disk from an Enzo dataset is now nearly 50% faster than in 2.X, and the memory overhead is considerably lower (as we get rid of many intermediate allocations.) Using yt to analyze Octree data such as RAMSES and NMSU-ART is much more straightforward, and it requires no costly regridding step.

Perhaps the best reason to want to move to 3.0 is that it's going to be the primary line of development. Eventually 2.X will be retired, and hopefully the support of Octree and SPH code will help grow the community and bring new ideas and insight.

How Can I Help?

The first thing you can do is try it out! If you clone it from http://bitbucket.org/yt_analysis/yt-3.0 you can build it and test it. Many operations on patch based AMR will work (in fact, we run the testing suite on 3.0, and as of right now only covering grid tests fail) and you can also load up RAMSES data and project, slice, and analyze it.

If you run into any problems, please report them to either yt-users or yt-dev! And if you want to contribute, whether that be in the form of brainstorming, telling us your ideas about how to do things, or even contributing code and effort, please stop by either the #yt channel on chat.freenode.org or yt-dev, where we can start a conversation about how to proceed.

Thanks for making it all the way down -- 3.0 is the future of yt, and I hope to continue sharing new developments and status reports.

EnzoTools Blog: Our New Blog

matthewturk@gmail.com (Matthew Turk) — Sun, 04 Nov 2012 22:09:04 +0000

Hi everyone! Welcome to the new yt Project blog. We've gotten rid of the old Posterous-based blog in favor of making it easier to include code, entries from anybody in the community, and to overall make it easier and clearer how to contribute.

So, to that end, we've moved to using a combination of pretty cool technologies to make it easy to blog and have your entry added to the blog.

For the blogging itself, we use Blohg, which is a mercurial-backed system. So all the blog entries are stored in a mercurial repository, on BitBucket (yt_analysis/blog) and instead of being in HTML or something, they're written in ReStructured Text (ReST) -- which is the same format that the yt docstrings and documentation are all written in. We're standardizing on ReST, which means to contribute to any of yt, you only have to learn one way to format your text. (Plus, ReST is super easy.)

To add a new entry, you just have to fork the blog repository and then issue a Pull Request. You can add the entry by creating a new file in the directory content/post, and it'll automatically show up with your name and the time you added it. Once your pull request is accepted, the blog will be automatically rebuilt and uploaded to the blog site (thanks to Shining Panda, which we use for our testing suite -- more on that later!) which lives inside Amazon's cloud.

But the best part is that this is all hidden behind the scenes. For all intents and purposes, you just need to add your text, issue a pull request, and it'll show up in a few minutes.

But here's the best part -- by converting to this system, we've also made it easy to include code samples using the IPython Notebook. A bunch of the yt developers have started using the IPython notebook for basically everything -- analysis, teaching, sharing snippets -- and we want to keep using it for everything. (If you take a look over at https://hub.yt-project.org/ you can see that we've started uploading Notebooks to the yt Data Hub, which then get displayed by the amazing NBViewer project by the IPython developers.) So, we made it easy to include a notebook here in the blog.

To include the notebook, you'll first need a copy of the NBConvert repository, which will also need to be in your PYTHONPATH. You may also need to install the "pandoc" project, but that's usually included in most Linux distributions and can be gotten with MacPorts. Once you've added that, just cd to the blohg directory and run::

python2.7 blohg_converter.py /path/to/your/notebook.ipynb

This will grab all the images and put them in the right directories inside the blog repository, add a new .rst file, and then you're set to go. Just run hg ci -A and you're good to go!

Because this blog is a bit new, we're still working through some kinks. Already as I've made a couple changes, the RSS feed has marked itself as completely updated; this is an error, so I'm trying to figure out what's going on and fix it up. So I apologize in advance if any other minor glitches happen along the way!

With this change in the blogging system, I think we've lowered the barrier to sharing with the community changes in yt, new features, and even showing old features using the Notebook. I'm really optimistic

And if you have something you would like to share -- a new paper you've written, something cool you've done (even if not in yt!) or anything else, go ahead and fork the repository and write up a blog post -- everything you need comes in the box!

EnzoTools Blog: Simple Grid Refinement.ipynb

matthewturk@gmail.com (Matthew Turk) — Tue, 30 Oct 2012 22:16:54 +0000

Notebook Download

Grid refinement

In yt, you can now generate very simple initial conditions:

In[1]:

from yt.mods import *
from yt.frontends.stream.api import load_uniform_grid
from yt.frontends.gdf.api import *
from yt.utilities.grid_data_format.writer import write_to_gdf

class DataModifier(object):
    pass

class TophatSphere(DataModifier):
    def __init__(self, fields, radius, center):
        self.fields = fields
        self.radius = radius
        self.center = center

    def apply(self, grid, container):
        r = ((grid['x'] - self.center[0])**2.0
         +   (grid['y'] - self.center[1])**2.0
         +   (grid['z'] - self.center[2])**2.0)**0.5
        for field in self.fields:
            grid[field][r < self.radius] = self.fields[field]

data = na.random.random((256, 256, 256))
ug = load_uniform_grid({'Density': data}, [256, 256, 256], 1.0)

yt : [INFO     ] 2012-10-30 18:11:48,715 Loading plugins from /home/mturk/.yt/my_plugins.py
yt : [INFO     ] 2012-10-30 18:11:49,025 Parameters: current_time              = 0.0
yt : [INFO     ] 2012-10-30 18:11:49,026 Parameters: domain_dimensions         = [256 256 256]
yt : [INFO     ] 2012-10-30 18:11:49,026 Parameters: domain_left_edge          = [ 0.  0.  0.]
yt : [INFO     ] 2012-10-30 18:11:49,027 Parameters: domain_right_edge         = [ 1.  1.  1.]
yt : [INFO     ] 2012-10-30 18:11:49,028 Parameters: cosmological_simulation   = 0.0
yt : [INFO     ] 2012-10-30 18:11:49,028 Parameters: current_time              = 0.0
yt : [INFO     ] 2012-10-30 18:11:49,028 Parameters: domain_dimensions         = [256 256 256]
yt : [INFO     ] 2012-10-30 18:11:49,029 Parameters: domain_left_edge          = [ 0.  0.  0.]
yt : [INFO     ] 2012-10-30 18:11:49,029 Parameters: domain_right_edge         = [ 1.  1.  1.]
yt : [INFO     ] 2012-10-30 18:11:49,030 Parameters: cosmological_simulation   = 0.0

In[2]:

spheres = []
spheres.append(TophatSphere({"Density": 2.0}, 0.1, [0.2,0.3,0.4]))
spheres.append(TophatSphere({"Density": 20.0}, 0.05, [0.7,0.4,0.75]))
for sp in spheres: sp.apply(ug.h.grids[0], ug)

yt : [INFO     ] 2012-10-30 18:11:49,035 Adding Density to list of fields

In[3]:

p = ProjectionPlot(ug, "x", "Density")
p.show()

Initializing tree  0 /  0  0% |                               | ETA:  --:--:--
Initializing tree  0 /  0100% ||||||||||||||||||||||||||||||||| Time: 00:00:00
Projecting  level  0 /  0   0% |                              | ETA:  --:--:--
Projecting  level  0 /  0 100% |||||||||||||||||||||||||||||||| Time: 00:00:01
yt : [INFO     ] 2012-10-30 18:11:53,889 Projection completed
yt : [INFO     ] 2012-10-30 18:11:53,894 xlim = 0.000000 1.000000
yt : [INFO     ] 2012-10-30 18:11:53,894 ylim = 0.000000 1.000000
yt : [INFO     ] 2012-10-30 18:11:53,895 Making a fixed resolution buffer of (Density) 800 by 800
yt : [INFO     ] 2012-10-30 18:11:53,914 xlim = 0.000000 1.000000
yt : [INFO     ] 2012-10-30 18:11:53,915 ylim = 0.000000 1.000000
yt : [INFO     ] 2012-10-30 18:11:53,915 Making a fixed resolution buffer of (Density) 800 by 800
yt : [INFO     ] 2012-10-30 18:11:53,934 Making a fixed resolution buffer of (Density) 800 by 800

We can even save them out to disk!

In[4]:

!rm /home/mturk/test.gdf

In[5]:

write_to_gdf(ug, "/home/mturk/test.gdf")

In[6]:

pf = GDFStaticOutput("/home/mturk/test.gdf")

yt : [INFO     ] 2012-10-30 18:11:56,370 Parameters: current_time              = 0.0
yt : [INFO     ] 2012-10-30 18:11:56,371 Parameters: domain_dimensions         = [256 256 256]
yt : [INFO     ] 2012-10-30 18:11:56,371 Parameters: domain_left_edge          = [ 0.  0.  0.]
yt : [INFO     ] 2012-10-30 18:11:56,372 Parameters: domain_right_edge         = [ 1.  1.  1.]
yt : [INFO     ] 2012-10-30 18:11:56,373 Parameters: cosmological_simulation   = 0.0

In[7]:

p2 = ProjectionPlot(pf, "x", "Density")
p2.show()

Initializing tree  0 /  0  0% |                               | ETA:  --:--:--
Initializing tree  0 /  0100% ||||||||||||||||||||||||||||||||| Time: 00:00:00
Projecting  level  0 /  0   0% |                              | ETA:  --:--:--
Projecting  level  0 /  0 100% |||||||||||||||||||||||||||||||| Time: 00:00:01
yt : [INFO     ] 2012-10-30 18:11:57,908 Projection completed
yt : [INFO     ] 2012-10-30 18:11:57,914 xlim = 0.000000 1.000000
yt : [INFO     ] 2012-10-30 18:11:57,914 ylim = 0.000000 1.000000
yt : [INFO     ] 2012-10-30 18:11:57,915 Making a fixed resolution buffer of (Density) 800 by 800
yt : [INFO     ] 2012-10-30 18:11:57,934 xlim = 0.000000 1.000000
yt : [INFO     ] 2012-10-30 18:11:57,935 ylim = 0.000000 1.000000
yt : [INFO     ] 2012-10-30 18:11:57,935 Making a fixed resolution buffer of (Density) 800 by 800
yt : [INFO     ] 2012-10-30 18:11:57,954 Making a fixed resolution buffer of (Density) 800 by 800

Over time, this functionality will expand to include cell-flagging, refinement, and much more interesting modifications to grid values.