JenningsPlanet.com Feed

Updates

2009-05-07T06:35:18Z

05.06.2009 Jet Propulsion Laboratory Open House 2009

I just returned from the JPL Open House. It was very cool. They are doing so many things related to remote sensing. Even the stuff they are doing for extra terrestrial endeavors benefits the Earth observation work. JPL has an entire branch devoted to Earth monitoring.Here is a slide show I put together from some pictures I took while I was there. Enjoy!

Cartography

2009-05-07T06:34:14Z

Cartographic Products

2009

Urban Forest Canopy of Sacramento (2009)

2008

1854 Sacramento Map (2008)

1950 American River Flood Map (2008)

2006

Digital Elevation Model of Sacramento (2006)

Urban Forest Canopy Map (2009)

The Urban Forest Canopy was generated from using the Feature Analyst 4.2 extension from Visual Learning Systems on ArcGIS 9.3. The primary imagery was high resolution QuickBird satellite imagery from July 2006. Independent Project for American River College Remote Sensing Course.

Urban Forest Canopy (letter) Urban Forest Canopy (22 x 22)

1854 Sacramento Map (2008)

33x44 (E Size, 14Mb PDF)

1854 Map of Sacramento based on a high resolution digital scan (B/W) from the Sacramento Historical Archives (www.cityofsacramento.org/ccl/history/index.html). The historical image was rectified to 2006 high resolution (6 in) digital orthophotography. The historical image is shaded with a brown color ramp; The Sacramento and American Rivers, parks, Sutter's Fort, railroads, ditches, and points of interest were digitized from the digital scan. Created while at the City of Sacramento.

1950 American River Flood Map (2008)

33x44 (E Size, 12Mb PDF) Hillshade 33x44 (E Size, 16Mb PDF)

1950 American River Flood Map based on scanned aerial photography which were rectified to 2006 high resolution (6 in) digital orthophotography. The blue shaded area represents the approximate extent of the flood. The current American River channel is shown in dark blue. The current levee system is shown in red. The estimate flow rate at the time of the flood was 180,000 cfs. The Folsom Dam nor the Nimbus Dam (upstream) were not yet completed. Fui Fang Thong completed the image rectification as part of a student internship program with the City. See the Student Projects Section for a summary of her work. Created while at the City of Sacramento.

Digital Elevation Model of Sacramento (2006)

(letter, 4Mb PDF) 34x44 (E Size, 24 Mb PDF)

BGR Color Shade (letter, 4Mb PDF) 34x44 BGR Color Shade (E Size, 24 Mb)

DEM Bare Earth Elevation (Detail). This is a detail of the inset from above showing the ancient American River corridor.

Digital Elevation Model of Sacramento based on 2004 LIDAR, 2 ft pixel spacing. The area detail shows the ancient course of the American River. Created while at the City of Sacramento. This data set was created using a combination of PERL, Surfer, Python, and ArcGIS Scripts.

Application Development

2009-05-07T06:33:43Z

Application Development

ArcGIS

Route Book Automated Map Production (Feb-April 2009)

Dynamic Map Template Designer (Feb 2007)

Accuracy Assessment (Feb 2006)

Field Map/Field Form Production (Aug 2005)

Python

Custom Python Scripts for ArcGIS Tools (Feb-April 2009)

Accuracy Assessment (Feb 2009)

Batch Kriging (Sept 2008)

Rental Inspection Program Monthly Parcel Master Build (Apr 2008)

Batch ASCII Text LiDAR to GRID (May 2005)

Other

Image Processing Application Development (in progress)

Opticks Plug-in Develolpment (2008)

Python Image Processing Application (2007)

ArcGIS Applications

I have written a number of tools and applications for various tasks at work,for my programming class, and as a consultant. I will have screen captures and possibly sample code or compiled routines for use. Check back for updates. If you use these samples, please give me and others credit. nate@jenningsplanet.com

Route Book Automated Map Production (Feb - April 2009)

This application solicits a Solid Waste service type and loads the respective routes into a listbox. The user then chooses one, many, or all of the routes to automatically generate 4 sets of maps that make up the map books for the selected routes. PDFs are created based on a pre-defined template. Approximatey 500 PDFs were automatically generated in 20 minutes.

Implemented

- VBA within ArcMap
- Custom user form for input
- Export to PDF function

Solid Waste Route Map Tool      Route Map Documentation

   Vicinity Map     Overview Map

Route Map 1 of 2 (Streets Only)     Route Map 2 of 2 (Streets Only)

Route Map 1 of 2 (Streets and Parcels)     Route Map 2 of 2 (Streets and Parcels)

Map Template Designer (Feb 2007)

This application provides a userform to dynamically generate a map of some common map sizes and add common elements such as title, legend, north arrow, title block for text elements, and date. Makes the page set up much easier by using a single interface. Developed for my GIS programming class.

Elements Used

- User Interface
- Page Layout
- Graphic Surrounds (Title, Date, Map Path, North arrow, scale bar, and graphic objects)
- Dynamic Map Frame setup, page size, font type and size

Map Template Desing Tool

Click the graphic to see the Map Template Design GUI

The user can select from a number of common page sizes. The GUI automatically shows the map frame dimensions based on a 1" margin, populates the current date, and provides some default values for a user to place a blank rectangle along the side of the map frame for a Map Legend.

The user can add a title and choose a particular font size and position it at a particular location. In addition, the user can add an MXD path that will show up on the lower left corner. The user can also add a North arrow, if desired.

Map Template Example

Accuracy Assessment (Feb 2006)

This program generates an Error Matrix to compute the statistical accuracy assessment elements for a land cover classification. The final output is in a DBF which can easily be brought into Excel and reformatted, if needed. Originally developed for my Remote Sensing Class, I also use this program in my GIS programming class to discuss table creation, joining, and the use of Search, Insert, and Update Cursors. (I also have a Python version of this program as well).

Elements Used

- Create Table
- Join Tables
- Search, Insert, Update Cursors
- Table Format
- Set/Get values from multiple tables and joined tables

Accuracy Assessment Tool

Click the graphic to see the Accuracy Assessment GUI

Accuracy Assessment Statistical Elements

- Categorical User's and Producer's Accuracy, High and Low Confidence Intervals
- Overall Accuracy
- KHAT

Error Matrix

The output error matrix is a dBase file and can easily be reformatted in Excel.

The error matrix columns represent the reference land cover class numbers.
The first column ("Class") of the table shows the classified land cover class numbers.

The bottom row contains the column total for each class (used in Producer's calculations).
The "Total" column contains the row total for each class (used in User's calculations).

User's and Producer's Categorical Accuracy Assessment values and Confidence Intervals are computed in the respective column.

Overall and KHAT values are computed in the far right columns.

Field Map/Field Form Production (Aug 2005)

This custom tool allows a user to select a neighborhood from a drop down list. The map layout will zoom to the extent of the appropriate neighborhood and perform a series of spatial and attribute queries to select residential addresses to inspect to verify the City of Sacramento's Master Address Database. The selected records are automatically output to a properly formatted Excel spreadsheet. This spreadsheet (many pages) is the field form used when field verifying addresses. The maps use a pre-defined template (E size, 33x44) that is also used in the field.

Elements Used

- Pre-load Combobox
- Spatial and Attribute Queries
- Zoom to Extent
- Output Selected Records
- Format and Export Selected Records to Excel file

Python

Custom Python Scripts for ArcGIS Tools (Feb-April 2009)

Two Python scripts were developed and linked to two separate custom ArcGIS Tools to process a street master file for an on board mobile emergency routing software. A custom ArcGIS Toolbox was created that stored the two custom tools.

Implemented

ArcGIS Custom Toolbox

Update Fields and Calculate Values Tool

- Add Standardized Fields to a file based geodatabase
- Query and Calculate appropriate values
- Appends changed features to master street file for further processing

Parse Road Classes Tool

- Perform a series of attribute queries on street types
- Write each of the different street type features to a new file geodatabase feature class

Accuracy Assessment (Feb 2009)

This script replicates the functionality of the Accuracy Assessment VBA version. The Python version is used in my GIS Programming class to demonstrate table creation, cursors, joins, and looping structures.

Elements Used

- Create Table
- Join Tables
- Search, Insert, and Update Cursors
- For and While Loops
- Error Checking

Batch Kriging (Sept 2008)

This script is being used to run 3 sets of Kriging operations on Califorina statewide daily precipitation and temperature data collected over a 4 year period that is organized by zip code. A kriging algorithm is used to generate a minimum, mean, and maximum surface of daily precipitation data and temperature data. Several million records of data exists for precipitation and temperature data for each year. The output of each of set of information is used in a spatial model to investigate West Nile virus vectors. This script was developed for UC Davis Medical School, Veterinary and Epidemiology Department research.

Implemented

- Personal Geodatabase
- Kriging algorithm
- Input/Output table template and creation

Rental Inspection Program Parcel Master Build (Apr 2008)

The Rental Inspection Parcel Build process is a set of 5 Python scripts to process a parcel master file that is usable to a 3rd party, hosted Case Management web application used by the City of Sacramento's Code Enforcement Department. These scripts are run on a monthly basis to update parcel data, flag exempted properties, calculate an estimated number of billed units, flags foreclosed properties, and creates a separate addresses per parcel table.

Script 1 - Update Parcel Related Spatial and Tabular data

GIS Processes: Copy Features, SDE Tables

- Delete and replace parcel points, parcel polygons, assessor's master table, assessor's appraisal table, assesssor's sales table, and master address file

Script 2 - Create Parcel Master File

Data Used: Parcel Points, Assessor Master file, Assessor Appraisal Table, Assessor Sales Table, Master Address File

GIS Processes: SDE Table Views, Spatial Join, Intersect, Attribute Query, Select by Location, Add Fields, Copy Features

- Query City of Sacramento Data from Sacramento County Parcel Data
- Spatial Joins and Intersects for overlay of various spatial data layers
- Create SDE Table Views with complex joins and attribute definitions

Script 3 - Calculate Fields

Data Used: Parcel Master file created above

GIS Processes: Attribute Query, Calculate Fields, SDE Tables and Table Views, personal geodatabase tables

- Query and flag Residential Properties based on land use
- Query and flag rental type properties based on ownership and tax exemption
- Query and flag exempted properties from Rental Inspection Program
- Calculate total units
- Calculate billed units based on exemptions
- Identify and flag foreclosed properties
- Create final Parcel Master File (personal geodatabase table)

Script 4 - Create Master Address/Parcel File

Data Used: Parcel Master file, Master Address file

GIS Processes: SDE Table Views, Attribute Query, Joins, Frequency, personal geodatabase tables

- Create a Parcel file that includes all unique addresses per parcel

Script 5 - Update Parcel and Low Income Housing SDE Table Views for Maps

Data Used: Parcel Master file

GIS Processes: SDE Table Views

- Update a parcel polygon file that flags low income housing and updates total units for low income housing and total billed units

Batch ASCII Text LiDAR to GRID Processing (May 2005)

This is a series of scripts (PERL, Surfer Grid, DOS .BAT file, and Python) to partition, create a surface, convert from ASCII Raster to ESRI GRID, then mosaic raw LiDAR XYZ ascii text information to generate a 2 ft DEM for the City of Sacramento.

This process took 2 full weeks of 24/7 processing on a Pentium III desktop computer. Over 10 billion LiDAR XYZ values were processed. Data management of various intermediate files was paramount.

PERL Script

Originally created by Berkley graduate students. Modified to accomodate the City's LiDAR XYZ text files.

- Processed and re-tiled ~2000 ascii files containing XYZ values
- Repartitioned LiDAR data into 2000 x 2000 m tiles

Surfer Grid Script

Originally created by Berkley graduate students. Modified to generate appropriate surface for City LiDAR data.

- Modified original surface generation algorithm to use the Surfer Grid Kriging algorithm. Tested and used appropriate Kriging parameters as needed.

DOS .BAT file

- Used to convert Surfer Grid format to ESRI ASCII GRID format

ESRI Spatial Analyst Mosaic Tool

- Used this tool to mosaic all GRID DEMs to a single DEM file (ESRI GRID format)

Other

Join_Table.py - This code was developed to demonstrate to my students the following (requires ArcGIS 9.1 or later):

The code is heavily documented to show thought process.

Create a Table from Scratch
Search, Insert, and Update Cursors
Join a Table (requires MakeFeatureLayer)
Write data to a new file

ArcIMS

XML Manipulation - Edit a Master File that is used for a number of internal/external applications for the City of Sacramento. Developed Spatial Queries, Thematic Symbology, SDE Layers

Java Script

Hacking as I go on this one. Primarily starting with available code, then manipulating it for my own purposes such as the Export to GPX Waypoint format that can be found in the Links/GPS

Many of the Google Maps you see on the website have been built from the base Google Map API with my twist and the use of the Google Maps Hacks book and other on-line resources

09.07.06

I am working on an Accuracy Assessment Program that will function within ArcGIS using polygon "Accuracy Assessment Samples", the Zonal Majority routine in Spatial Analyst, and the classified imagery.

Image Processing Application

Opticks Plug-in Development (R&D since 2008)

Opticks is an Open Source Digital Image Processing software developed by Ball Aerospace and encourages outside parties to contribute improvements and add enhanced functionality. The code is written entirely in C++ using Windows OS and Visual C++. The "free" version of Visual C++, v. 2005 can be used to further develop Opticks.

I have started investigating Opticks as a "free" software to supplement my Remote Sensing class so that students can gain more hands on experience with the digital image processing concepts learned in class. I am hoping to develop the following plug-ins to Opticks when time permits:

Edge Detectors

Image Filters - High Pass, Low Pass, statistical

RADAR Processing - Slant-to-Ground Range correction, Texture Analysis

Accuracy Assessment

Image Classification

Signature Evaluation

Python Image Processing Application (R&D since Nov 2007)

The Python Image Processing Application is a research project of mine that I periodically work on that I use to develop new concepts for my GIS programming class and developing Python code for use in other aspects of my professional career.

11.15.2007

Mean Filter Optimized.py

I have done a little optimization on my previous code. The same 6 band 400x600 image now runs in ~10 sec v. 5 min. Thanks to Chris Barker at NOAA who provided with numpy arrays and an array method for processing a filter across an image. Be aware that a very large image will take a very long time. 6x3000x4000 took ~ 2 hrs and maxed my page file out--probably has something to do with large arrays.

11.05.2007

Mean Filter.py

Here is a function I wrote that will be part of my Image Processing application. This is my first attempt at writing a "Python" function using gdal for "Python", Numeric (old NumPy/SciPy), and a window kernel. Enjoy. Please submit any comments or suggestions/methods for improvement to nate@deltateck.com

06.12.2007

I have started writing some shell code to run some basic image processing routines. The code is written entirely using the wxPython, NumPy, "Python" Image Library, SciPy, MatPlotLib, modules ("Ptyhon" for GUI apps). In the future I hope to inlcude the open source GDAL libraries under "Python."

Image Processing wxPython_v. 0.0.1_03.18.2007.py (update 03.18.2007) This is very early code written in "Python" using the "wxPython" modules that are part of "wxPython". I hope to continue work on this as time permits for my remote sensing class. This is to be used as a teaching aid to students and myself who cannot afford the commercial software, but want it a little more functional than other freeware. The "(Python) Image Library" is expected to be used.

Currently there are no functional routines, except the Help-->About.

I will include basic image processing functionality as I can get to it or others would like to contribute. Email me code/ideas at: nate@jenningsplanet.com

I would like to include:

Viewing - simple loading of remotely sensed imagery, zoom in/out, pan, zoom to extent
Utilities - subsetting, file conversion
Image Enhancement - Contrast Stretch, Principle Components
Image Classification - ISOData (unsupervised), MaxLike (supervised)
RADAR - slant-to-ground range, texture analysis

KML

This is a new addition to the website based on something I was working on at the City and with the Google Map community. I would like to thank the assistance of Mike Williams for his input and vast knowledge of Google Maps and the API.

What is shown here are boundaries of some areas within the city of Sacramento. A KML File was created using a KML converter found at the ESRI Site (1.18.07), then used within the Google Maps. There is communication between my site and Google Maps to render the map you see here. This version does not render the label points automatically. I will have to program this until other improvements come along. The alternative version of this map that does show the labels and a legend can be found here: KMLonGoogleMaps

GIS

2009-05-07T06:32:51Z

GIS is more than just managing spatial data; it requires the ability to generate useful information from the data.

GIS Analysis

Overlay
Data Integration
Data Creation
Update and Modification
Categorization
Automated Processing
Modelling
Reporting
Statistics

Application Development

ArcObjects
Python
VB Script for ArcPad

Cartographic Output/Production (digital and Hardcopy)

Site Maps
Planning Maps
Field Maps
Map Atlases
Map Books

GIS Software

ArcGIS (ArcGIS, ArcIMS, ArcSDE, ArcPad)
QGIS, Postgres

Remote Sensing

2009-05-07T06:24:22Z

Today's world of spatial data and analysis requires solid knowledge of remotely sensed imagery and digital image processing.

Below summarizes my knowledge and professional experience in Remote Sensing Technology.

Sample Projects

Urban Forest Canopy

Urban Forest Canopy (April 2009)

Data Acquisition (Satellite, Aerial)

Knowledge of Vendors
Vendor Selection
Technical Specifications
Knowledge of Sensor Sources

Image Processing

Satellite, Aerial Photography, RADAR, LIDAR, Hyperspectral
Integration with GIS
Rectification
Classification
Image Fusion (Sensors, Resolutions)

Urban Forest Canopy (April 2009)

I derived the Urban Forest Canopy from QuickBird (July 2006, 2ft (0.6m)) imagery using Feature Anlyst on ArcGIS. I developed a protocol for my Remote Sensing students to follow for a segment I was teaching on high resolution image processing techniques.

Implemented

- Feature Analyst 4.2 on ArcGIS 9.3
- Created protocol for deriving Urban Forest Canopy

Urban Forest Canopy Protocol Excerpt

QuickBird Subset (13 Mb)

Urban Forest Canopy (letter) Urban Forest Canopy (22 x 22)

Software

ERDAS Imagine
Feature Analyst
TNT Mips
Grass
ESRI