Tag Archives: ArcGIS

Mesonet Collects High-Accuracy Weather Data for Mapping and Data Visualization Applications

Posted on by
Reply

University at Albany Program Hosts Third Annual Symposium September 13-14

Background

In April 2014, the State of New York in collaboration with the Division of Homeland Security and Emergency Services funded the University at Albany to design, install, and operate the NYS Early Warning Weather Detection System. The centerpiece of the system is the New York State Mesonet (NYSM) network of 126 weather stations across the state, with at least one site in every county and borough. Each of the Mesonet’s 126 weather stations collects observations of surface temperature, relative humidity, wind speed and direction, precipitation, solar radiation, atmospheric pressure, snow depth, and soil moisture and temperature at three depths (5, 25, and 50 cm). Each site is also outfitted with a camera that collects still images.

In addition, several sub-networks of specialty sites have been deployed. A Profiler Network of 17 sites, which leverages LiDAR technology, provides additional atmospheric data in the vertical (up to 6 miles above ground); a Flux Network of 17 stations monitors the surface energy and a Snow Network of 20 sites which measures snow water content. Operated by the University at  Albany, the New York State Mesonet collects, archives, and processes data in real-time every five minutes, feeding weather prediction models and decision-support tools for users across the greater New York region. The NYS Mesonet implementation is part of the larger National Mesonet Program. NYSM is a cross-disciplinary team supporting a standard of excellence in delivering high-quality weather data across the Empire State.

The Mesonet network includes stations in all 62 New York State counties including the five boroughs of New York City

Metadata is provided for each site including lat/long, local landscape characteristics, pictures, elevation, soil classification, among other items.  Adjacent variables, such as obstructions, are reflected in a World Meteorological Organization (WMO) rating which indicates that the site’s surrounding environment may cause uncertainty in the data.

Detailed metadata is available for each station across the Mesonet network which can be used in supporting local weather-related mapping and analysis applications

Data and Geospatial Products

NYSM utilizes these high quality weather data to create end products designed for use by government, academic, and with commercial partners. NYSM supports solving real-world problems in agriculture, emergency management, energy, transportation and protecting public health. Much of the mapping and data visualization products on the Mesonet site is the responsibility of Nathan Bain, Sr. Software Engineer, NYS Mesonet, University at Albany, SUNY.  “The core set of software products I use in publishing content on the website include OpenLayers, python scripting, REACT for website development, and Highcharts for interactive time-series plots and line graphs.”

According to Bain, frequent users of the data are the National Weather Service, New York State Department of Health, New York State Department of Environment of Environmental Conservation, a variety of forensic meteorological companies, and private citizens.  Forty-four maps are updated every five minutes as presented on this page.

24-hour temp change is one of the many maps which is refreshed every five minutes on the Mesonet website

Mesonet field station data can be requested by completing and agreeing to the terms found in the Data Request Form. Data can be delivered in either NetCDF (network Common Data Form) which is a format for storing multidimensional scientific data (variables) such as temperature, humidity, pressure, wind speed, and direction. Each of the variables can be displayed through a dimension (such as time) in ArcGIS by making a layer or table view from the netCDF file. Data can also be delivered in normal CVS  format. Today’s weather statistics are available here. One interactive mapping application which Mesonet publishes “live” is its Transportation app showing the weather variables from sites along the NYS Thruway.

While many of the Mesonet Thruway weather stations are located in western New York near Buffalo, real-time data is collected as far east as Oneida.

September 2023 Conference

The Third Annual New York State (NYS) Mesonet Symposium will be held on 13-14 September 2023 at UAlbany’s ETEC building. Similar to previous symposiums, this will be a chance for people from academia, government, and the private sector to present on research and operational applications of the NYS Mesonet data. While in-person participation registration has been closed, those interested in participating via Zoom can use this link and a copy of the September agenda including presenters and participating organizations is available here.

Contact:

Nathan Bain
Software Engineer
NYS Mesonet
University at Albany
nbain@albany.edu

Summer Reading: The Drones Are Amongst Us

Posted on by
Reply

Transformation geospatial technology is emerging as a “must have” in government GIS programs

Last month Governor Kathy Hochul announced the completion of the Skydome facility at the Griffiss Business and Technology Park in Rome, Oneida County. Located in a former airplane hangar, the indoor facility is the largest indoor drone test facility in the nation and a major investment in New York’s Unmanned Aircraft Systems (UAS) Test Site.  This investment adds to the already impressive UAS and high-tech ecosystem in the Empire State centered along the Syracuse-Rome corridor.  Its no news at this point that UAS are a transformation technology in the GIS and geospatial space with cutting-edge research and development going on right here in our backyard.  The Northeast UAS Airspace Integration Research Alliance (NUAIR) website provides a great example of this growing technology space which includes government, industry and academia.

The governor’s announcement made me go back looking for my notes on the Systems and Technologies for Remote Sensing Applications Through Unmanned Aerial Systems (STRATUS) 2022 Conference held May  23-25 at SUNY ESF which I wanted to participate in and attend, but the scheduling didn’t work out.   The 2022 show was the fifth STATUS conference which brought together more than 100 professionals from across the United States and Canada.  New York’s geospatial community, including those from a host of academic institutions and local and state governments, was well represented in the nine technical sessions which included both oral and poster presentations covering topics including mapping and surveying, agriculture and forestry, flood and water quality monitoring efforts, hardware and software, and flights and operations. Great homegrown UAS research work coming out of Cornell, Rochester Institute of Technology, SUNY Binghamton, Hobart and William Smith College, and SUNY ESF.   One of the keynote speakers was Ken Stewart, Chief Executive Officer of NUAIR who I interviewed as part of a 10 Questions column last fall.  The STRATUS 2022 program guide, including abstracts and presentations can be found here. 

And for those coming out of college or those looking for a mid-career change – its a great job market.   If not a great job skill to augment any current geospatial position.    Commerical UAV News reports the top five drone pilot industry markets (in order) include:  mapping and modeling, thermal imaging, real estate photography and videography, inspections, and movies and television.  Another publication predicts that by the year 2025, at least 100,000 jobs will be created for drone pilots. (btw – while putting together this article, came across an update on New York State drone laws and regulations as of July 21, 2022).

Seemingly now present in every significant geospatial data collection effort at all levels of accuracy and need, drone usage will grow exponentially throughout government in the years ahead.  As governments continue to remain concerned over drone liability, privacy, and cybersecurity issues,  industry and consulting firms will continue to provide the bulk of drone and UAS operations.  This may change slowly over time, but the near future work will be supported largely by the non-government workforce.  One exception to this will be in the public safety arena where government staff will continue to be aggressively trained and licensed internally.   

In other areas it will be most likely specific drone consulting firms or engineering companies will step-in and provide drone/UAS services.  Many engineering companies have already  positioned themselves well in supporting government geospatial projects by building GIS capacity and offering a wide range of services skillsets – both on the desktop and in the field.  And drones are a perfect fit as part of their surveying and mapping consulting work particularly for smaller to mid-size governments with limited or no technical staff.      These same engineering firms now offering full service GIS programs including the managing of online products such as ArcGIS Online to data warehousing.  Particularly advantageous to local governments as datasets get larger and larger as the televising of infrastructure systems (sanitary, storm, potable water, etc), above ground 3D modeling, building information models (BIM), and high resolution imagery continue to be captured and developed in municipal governments.  Continued advancements by the major software companies such as ESRI and AutoCAD, and their business partners,  on the integration of drone generated data such as lidar only extends the ease of use and accessibility of drone/UAS technology for government organizations.

And with high accuracy mapping and surveying being one of the most common applications of drone/UAS use, particularly in context of the quality of imagery captured and increasingly larger geographic footprints of data being collected, might it lead to  the discussion as to how the emerging drone/UAS technology augments larger publicly – funded efforts such as the USGS 3D Elevation Program (3DEP)?  While the 3DEP program covers much larger geographic areas and is plane-based, is there a way to add smaller project areas which are drone/UAS  based and meeting appropriate federal lidar specifications to public data warehouses and inventories such as the U.S. Federal Mapping Coordination website?  Expand the State elevation library to include locally collected data?  There is so much elevation data being collected across the state by so many sources.

Summary

Government GIS/geospatial practitioners will do well to follow and participate in technology specific conferences and associations such as those represented at STRATUS 2022.  Many of which are becoming more accessible by being available online.   Industry specific conferences in engineering, public works, public safety – even agriculture – now serve as an excellent source of specific applied geospatial technology tools.  And drone/UAS technology is a common thread among them all. 

Coming to a geospatial application close to you soon. 

Hard Copy Maps Still Rule: NYS Agricultural Districts

Posted on by
Reply

The art form is still alive as counties design and create maps as part of the Agricultural District Certification process

I’ve written about agriculture in New York State before in context of the farming community continuing to evolve with the adoption of geospatial technologies. With farmland covering nearly 20% of the state’s geographic footprint, its easy to see why geospatial tools are so increasingly important in managing the state’s prime agricultural lands.  And one does not need to look far to see the advancements of new geospatial technologies in New York State agriculture, particularly with the recent Systems and Technologies for Remote Sensing Sensing Applications Through Unmanned Aerial Systems (STRATUS) conference May 23-25, 2022 at SUNY ESF in Syracuse. Several papers and even dedicated tracks of papers and presentations focusing on UAV/drone technology in agriculture.

And it’s big business, too. A November 2021 article published by NY Business Agribusiness Column: Economic Impact of Agriculture in New York State stated that “agriculture in New York State contributed $43.6 billion in total industry output, 160,100 jobs and $12.3 billion in gross domestic product across the state economy.” It’s a rather amazing article with many more figures on how agriculture impacts the statewide economy, The article is based on the work and research of Todd Schmidt, an associate professor with the Charles H. Dyson School of Applied Economics Management at Cornell University.

Which makes it all the more interesting in 2022, even with the fledgling agritech applications and the almost complete conversion and automation of our mapping, cartography, and business/financial systems, the certification of agricultural districts in the state still involves the use of an iconic piece of mapping and cartography: The hardcopy map. Yes, while the digital data/products in producing the hardcopy maps are used and repurposed in many useful and productive ways across the state, it is still the hardcopy map which is used as part of the agriculture district certification process by NYS Department of Agriculture and Markets (AgMkts).

Background

Since 1971 , the Agricultural Districts Law, Article 25AA of the Agriculture and Markets Law (AML), as administered through (AgMkts), has reflected state and county level efforts to preserve, protect and encourage the development and improvement of agricultural land for the production of food, fiber and other agricultural products. As districts are created, or modified during renewal, maps are generated, based on tax parcels included in the districts. Each county is responsible for creating maps to meet the requirements of the Agricultural Districts Mapping Program. Supporting AgMkts in this regard is Cornell University’s Institute for Resource Information Sciences (IRIS) has helped facilitate the mapping program since 1977. Its primary responsibility is to support agricultural district mapping efforts by reviewing, distributing, and archiving agricultural district maps, and to provide consultation services. Automation of the statewide district maps began in earnest In 1996 when the entire set of NYS Agricultural District maps in the collection of Cornell IRIS (formally CLEARS) was converted to digital format. This was done by shipping blueprint copies of the maps to the NYS DEC for scanning.

Article 25AA references maps several times in Section 303 as a required deliverable for certification of a district. Specifically the text includes “The district review plan shall include a description of the district, including a map delineating the exterior boundaries of the district which shall conform to tax parcel boundaries”.  Agricultural districts across New York State are reviewed and certified on an eight-year cycle.

The Map

It’s a real treat anymore to show up at a meeting or discussion and be given the chance to review a traditional, well designed hard copy map. Increasingly becoming rarer and rarer. And just think how much less demand there would be for hardcopy maps if it wasn’t for the ubiquitous use and sharing of the PDF file format? For so many years, the 1:24,000 map was one of the most commonly used map design and layouts following in step with the classic U.S. Geological Survey 2000-scale quadrangle grid series. And yes, it just so happens the required base map standard for the agricultural district maps, as outlined in the Agiriculture Districts Program Mapping Checklist prepared by IRIS, is 1:24,000. According to Diane Ayers, who provides staff support to the mapping program, “IRIS recommends either the NYS DOT 7 ½ minute planimetric map series (available in either raster or vector format) as the base map for individual County mapping products.”

The 32-page document consists of written requirements covering topics such as, but not limited to, map content including roads, civil boundaries, and water features, as well as traditional map items such as north arrows, keys and legends, scale bars, and the bounding geographic coordinate grid.  Documentation on the source of base map data, other data layers, year of parcel data, and geographic reference information (datum, projection, etc) is also recommended.   It also includes several pages of examples covering a wide range of other hardcopy cartographic principles including annotation, color selection, and symbology.  Many items which cartographers and map makers have greater flexibility in controlling when using desktop client software vs. web mapping cartographic tools. 

Appendix 1 in the Mapping Checklist document shows examples of how to render the district boundary as the most prominent feature on the map

While counties send all review materials associated with the eight-year review, including tax parcel information, directly to AgMkts, the county produced hardcopy map and shapefiles are sent directly to Ayers.   She reviews the maps for consistency with the mapping checklist and then forwards to AgMkts and uses the shapefiles to maintain a consistent a set of statewide district maps which are available via CUGIR.  The statewide coverage she maintains based on this relationship and data exchange with the counties can be found here.

The 2022 Ontario Agriculture District Map as produced by IRIS. Shapefiles and documentation received from counties enables IRIS to maintain a statewide agricultural district coverage.

Sheri Norton is the GIS Coordinator for Ontario County, located in the Finger Lakes region renown for viticulture and other agricultural products. She has been through three district reviews including a large countywide consolidation in 2021. “Agriculture District mapping is a big effort with partnership between our Planning, Real Property and GIS Program staff”, she says, “but with my previous experience in producing the mapping products for IRIS and becoming more comfortable with ArcGIS Pro, the map series and formal reports are now easily updated for the next review cycles.  The ArcGIS client is all I use.”   Which is quite an accomplishment, as keep in mind many final cartographic products are often “touched up” before final publication with other desktop publishing software such as Adobe Illustrator. The certification also includes generating a list of all tax parcels in the district(s) based on the county assessment roll.

Sheri Norton’s 2021 Ontario County Agriculture District One map designed and produced entirely with desktop GIS software.  This same map was provided to AgMkts as part of the review and certification process

Summary

Hard copy maps are as close to GIS comfort food as we can get.   And here in New York is refreshing to see how this important piece of our cartography past is still being used in a government regulatory program.  Holding its own against all of the advancements – and acceptance of digital web mapping.  Kudos to GIS professionals across the state who are still producing hardcopy maps and helping the product remain relevant as part of the profession.

Agricultural Districts protect and promote the availability of land for farming purposes.  For informative overview of this important Empire State Program review this presentation.

Contact

Diane Ayers
Data Manager and Educator
NYS Agricultural Districts Mapping Program
Cornell IRIS
1015 Bradfield Hall
Ithaca, NY 14853
dag10@cornell.edu

Sheri Norton 
GIS Coordinator, GISP
Geographic Information Systems Program
Ontario County Information Technology Department
70 Ontario Street
Canandaigua, NY 14424
Sheri.Norton@ontariocountyny.gov

Jeff Kehoe 
Farmland Protection Specialist
Department of Agriculture and Markets | Division of Land and Water Resources
10B Airline Drive
Albany, New York 12235
jeffrey.kehoe@agriculture.ny.gov

10 Questions: Neil Curri

Posted on by
Reply

Geospatial technologies are found in so many organizations and on so many levels.
What once was a fledging technology based on desktop client software now
encompasses web technology, mobile and handhelds, increasingly the engineering
software toolbox, and hordes of geospatial data of all types and quality. And yes, the
ageless desktop clients lives on. It’s not often to find someone who is able to navigate
the many pieces of the technology for multiple organizations.

Neil Curri, who lives in the mid-Hudson River Valley community of Lloyd, is involved in a
wide range of geospatial efforts encompassing a diverse number of organizations over
the past several years. While technically currently employed by Partridge Venture
Engineering, Neil also serves as a GIS Academic Computing Consultant at Vassar
College in Poughkeepsie and has served in local government geospatial efforts as well
as the Dutchess County Cornell Cooperative Extension.  We recently communicated to discuss his past and current GIS contributions to the profession.

eSpatiallyNewYork: Let’s go back to the beginning – when and how did you get started
with geospatial technology?

Curri: After working in IT for a few years after college, I decided to go back to school to
refocus my career on something in the environmental field. GIS seemed a logical fit. I
found an Environmental Science M.S. program that also offered a GIS certificate at the
University of New Haven. The summer before I went back to school, I got a hold of a
copy of ArcView 3.2 and made a couple of maps and a website for a local watershed
group. The environmental applications were obvious.

eSpatiallyNewYork: In your capacity at Partridge Venture Engineering (PVE), what kind
of GIS applications or tools do you use?

Curri: Mostly ArcGIS Pro (and ArcMap for legacy projects – still transitioning…), some
GPS, and some limited drone work. We’ve used ArcGIS Online for a handful of projects
where a web mapping application was an appropriate deliverable for the project and
client. I’ve had to build a couple of small databases to organize environmental data
streamlined for mapping. My position is within the company’s Environmental division,
and there’s a need for map figures in every phase of a project – most of our work is site investigation and remediation. Mapping-grade GPS provides enough accuracy for ourpurposes, but we do use differential GPS and post-processing to get the best accuracy we can with our hardware. We’ll collect drone photos when existing aerials are
insufficient or outdated. I’ll either georeference them for any site figures we need, or
occasionally I’ll set up a flight plan and use drone mapping software to get a composite
aerial of a large area or use the 3D capability to estimate soil stockpile volumes or
excavation pit volumes.

eSpatiallyNewYork: You are the GIS Lab Manager at Vassar College. What does this
involve and some of your duties

Curri: My official title at Vassar is GIS Academic Computing Consultant. Duties include
mainly instructional support – advising faculty, staff, and students interested or engaged
in using GIS for teaching and research. For example, students in a Geomorphology
course were recently working on a GIS lab exercise that involves mapping changes in
glacial lake levels and creating an elevation profile to get channel dimensions and
calculate flow volume. This lab is one of several exercises for the course that were
originally written for ArcMap, which I migrated to ArcGIS Pro. I also work with the
college’s IT department staff to maintain GIS hardware, software, and licensing. CIS are
technically the “lab managers”, but we work together to keep things running and provide
capacity as GIS use grows on campus. With an enrollment of about 2,400 students,
Vassar is a small college. But GIS use is reaching the point where I think the
administration will expand licensing campus-wide for the next academic year, which will
significantly reduce the time I spend managing ArcGIS Online accounts.

eSpatiallyNewYork: Any particular area students seem to be most interested in in
applying GIS technology and concepts?

Curri: While the two core GIS courses – Cartography and Spatial Analysis (I
occasionally adjunct instruct either of these) – are taught out of the Geography and
Earth Science department, many students who take the course are Urban Studies
majors, who are very interested in social justice issues. But there are also many
Environmental Studies students who take the courses, and they tend to be interested in
ecological research and conservation. Neither of these courses (nor both of them
together) necessarily prepare students for a career in GIS, but students who take these
courses are sought after by faculty and staff for research and other projects (like the
Poughkeepsie Natural Resource Inventory or research projects at the Collins Field
Station on Vassar Farm) and they are more likely the ones who will pursue GIS as a
career and/or go on to graduate school to study GIS or a GIS-related field.

eSpatiallyNewYork: Tell us something about Vassar GIS students/grads,
YouthMappers, and Society for Conversation GIS

Curri: Hudson Valley Mappers is the Vassar-based chapter of YouthMappers, an
international organization supported by USAID. Two former students, one of whom now
works for USAID supporting YouthMappers, began the chapter in the Fall of 2018.
Students involved in HV Mappers are passionate about using GIS as a resource for
communities, both locally and around the world. The group coordinates local community
mapping events, usually involving some field data collection and partnering with one or
more local organizations, as well as organizing mapathons in which students participate
in Humanitarian OpenStreetMap Team (HOT) mapping projects

eSpatiallyNewYork: What was your geospatial roll and contribution with the City of
Poughkeepsie Natural Resource Inventory?

Curri: I mentored several student workers who assembled the maps created for the
report and online content and I ended up editing some sections of the report with Jen
Rubbo, Director of the Environmental, who led the project. Part of the it involved
inventorying and inspecting Ash trees for impacts by Emerald Ash Borer. I worked with
students and staff using previous tree inventory data by the City’s Shade Tree
Commission as the starting point, set up an ArcGIS Collector project for the field data
collection, and helped organize the information into a public-facing web map. That data
was used by the City to identify trees that presented a potential hazard and remove
them. There’s a short video describing that part of the project on the City’s website.

eSpatiallyNewYork: You worked at Dutchess County Cornell Cooperative Extension
between 2007-2015. What is your involvement there today?

Curri: I was recently asked to serve on an advisory committee for Dutchess County Cornell Cooperative Extension’s pending update of the County’s Natural Resource
Inventory. I worked on the last revision several years ago while at CCEDC, which was
an update of the previous version conducted in 1985 by the County’s Environmental
Management Council. A former intern and a volunteer I mentored there both went on to
graduate school and made careers for themselves in GIS – one is now with the NYS
GIS Program Office and the other is on staff at CCEDC.

eSpatiallyNewYork: You currently serve as Chair of the Town of Lloyd Environmental
Conservation Committee. What’s your sense of the understanding and acceptance of
GIS concepts by citizen and community groups at this level in the Hudson River Valley?

Curri: Though I may not be totally sure, but I think most citizen and community groups
in the Hudson Valley – at least any that have an interest in environmental conservation,
planning, or community development, or anyone volunteering for a town committee or
board – have some sense of what GIS is and that it has value. But there is a lack of
capacity available to citizen and community groups, and consultants like me cost
money, and budgets are limited, so I think exposure to GIS and understanding how it is
or could be useful to them is still limited. Lloyd is one of few towns in the Hudson Valley
with some in-house capability, which predates my involvement.

eSpatiallyNewYork: Some of your community outreach work and involvement has led
to having the DEC Hudson River Estuary Program support an intern on one of their
funded projects. Tell us about how that developed.

Curri: A former student should get some credit for this. While we were finishing up the
Poughkeepsie NRI, she was taking the web mapping class, which is a half-semester
class taught occasionally taught by Prof. Mary Ann Cunningham. Mary Ann connected
her with us, and for her class project the student published the NRI layers to ArcGIS
Online and assembled them in nested web apps to combine functionality. Volunteers
working on NRI’s in two other Hudson Valley towns, Union Vale and Gardiner, saw the
Poughkeepsie online map and wanted one of their own. Interested interns came forwardand used the Poughkeepsie NRI map as a template to assemble those. Based on thesuccess of these, the DEC Hudson River Estuary Program then contacted me to inquireabout assembling maps for the Town of Gardiner’s Community Preservation Plan
(CPP). We found an interested student but it ended up being more simple for PVE to
assume the contract and hire the student. He and I are working on this project now.

eSpatiallyNewYork: What’s been your involvement in statewide GIS activities?

Curri: I’ve been a member of the NYS GIS Association for a few years, but not very
active.I used to attend NEARC regularly before my partner and I started a family five years ago. About two or three years ago, some folks from a few land trusts in the
Hudson Valley started up a local chapter of the Society for Conservation GIS. I have
missed only one or two of these meetings since then. It’s a great group of people and I’dlike to be more involved – after this semester is over!

eSpatiallyNewYork: Have a particular GIS project you worked on or produced that
would like to highlight or brag about just a bit?

I think I’ve already done some of that here, but I’ll mention getting to work with Vassar
Professor Emeritus Dan Peck on the maps for his book, Thomas Cole’s Refrain.

There’s some work for litigation projects at PVE I’d like to highlight but not permitted top
do so. Most of my projects I don’t really consider “produced” by me, as I am often
working with a subject matter expert, a geologist, a professor, or enthusiastic student, a
client, or community partner who’s more or less guiding the project, and I’m just figuring
out what geospatial product or tool fits their needs.

Contact:

Neil Curri, GISP
GIS Analyst, PVE, LLC
Vassar College Academic Computing Consultant
necurri@vassar.edu | 845-437-7708 | 845-664-2100 (mobile)

Mapping a New Economy for Equality and Sustainability

Posted on by
Reply

New York City nonprofit uses GIS to help support cooperative and community-led development projects as well as advocacy campaigns

Introduction

Nonprofits are one of my favorite areas to cover in the geospatial space.  More than not their geospatial efforts operating on a shoestring budget, relying mostly on filtering through open data portals and mashing everything together to create visual products with open source software or via discounted vendor supported nonprofit software licensing agreements. Its not uncommon to find the person(s) doing spatial analysis and map making having been self-taught.  A lot gets done with a little.

One such organization is the New Economy Project (NEP) located in New York City.  NEP works with community groups to build a new economy that works for all, based on principles of cooperation, democracy, equity, racial justice, and ecological sustainability.  Their focused campaigns are carried out in concert with community, labor, civil rights and social justice groups on initiatives such as:

  • Public Bank NYC, a municipal public bank campaign rooted in economic and racial justice. Through public banking, NYC can divest from Wall Street banks, reinvest public deposits to support equitable and cooperative development.  
  • Advancing Community Land Trusts, through co-leadership of the NYC Community Land Initiative (NYCCLI) and one-on-one work with community groups seeking to organize community land trusts. 
  • Advancing NYS Community Equity Agenda, calling for transformation of the fundamentally unjust social, political and economic systems. The Equity Agenda puts racial and economic justice front and center and addresses long-standing inequities that play out at neighborhood and regional levels. 
  • Growing NYC’s Cooperative Economy through coalition-building and policy change that supports and brings together community land trusts; worker, housing, financial and food co-ops; and other models of cooperative, community-led development.

GIS and Mapping

Behind all of the mapping at the New Economy Project is Ben Hagen who currently serves as Senior Research and Communications Associate.  “As much of our work focuses financial services, a considerable amount of our data comes from the Federal Deposit Insurance Corporation (FDIC)” Ben notes, “and more specifically from the Summary of Deposits section.”   Ben does all of his mapping and data analysis with ArcMap having learned the craft as part of his graduate work in Urban Planning at New York University.  Other data sources frequently used by NEP include the U.S.Census Bureau, New York State Department of Financial Services,  New York State Office of Court Administration, and NYC OpenData. 

NEP uses use ESRI’s World Geocoding Service to support most of its address mapping though in the past had also used the city’s LION file. In addition to ArcGIS, the main other data-related software NEP uses is open source software R for cleaning data, especially when preparinfor for mapping. Hagen likes R software because it runs on text-based code and is great for replicating processes with minimal opportunities for introducing errors, as well as for easily sharing with others for a second look at methodology.  

Samples of NEP mapping products include (click image for map download).

(Left Image): Bank branches are not distributed and made accessible evenly across the city.  This map depict the relative absence of bank branches in  the Bronx, and the resulting concentration of high-cost alternatives such as check cashers and pawn shops. This is a series of maps that NEP updates every year covering most (if not all) of the city’s neighborhoods.  (Right Image): NEP creates maps showing the concentration of residential foreclosure risk in NYC communities of color. This uses data from the NYS Dept. of Financial Services, based on notices that mortgage servicers are required to send to delinquent borrowers 90 days before initiating a foreclosure action. While not all of these notices result in an eventual foreclosure, it provides a good indication of where the risk of home foreclosure is highest in the city.

(Left Image): Using data from the NYS Office of Court Administration, this NEP map highlights the concentration of debt collection lawsuits in communities of color and is included in this report prepared for AARP New York.  For additional maps, graphics and YouTube videos on this project visit this web page.  (Right Image):  Another map showing the relative absence of bank branches in NYC’s communities of color. Looking at this annually, one can see changes as gentrifying neighborhoods gain branches, but the overall pattern is a long-standing and remarkably stable one. There are actually six zip codes in the city, ranging in population from about 20,000 to 94,000 and all in communities of color, that have no bank branches at all.

Other examples of NEP research and mapping products can be viewed here.

Summary:

“The concept of “place” has always been at the core of how we approach our work” notes Hagen.  “In NYC, people feel very connected to their neighborhoods, and are acutely aware of how much the communities they call home shape their day-to-day lives. There’s no better way than maps to relate to people’s experience of their physical environment, and to help contextualize those environments within the larger, systemic frameworks that we are so often fighting to change.

Clearly, GIS is both an important business and commuication tool for NEP.

Contact:

Ben Hagen
Senior Research and Communications Associate
New Economy Project
121 W. 27th Street #804 | NY, NY 10001
212.680.5100, ext. 221 | neweconomynyc.org

 

 

 

 

 

Developing and Applying the Geographic Aggregation Tool (GAT) at NYS Department of Health

Posted on by
Reply

“R”-based software application is used by public health researchers and data analysts

While often much of the Empire State GIS software discussion evolves around the use of the ESRI platform, there continues to be a steady dose and discussion of alternative geospatial client and web-based products. For example, staff at the New York State Department of Health (NYSDOH) use ArcGIS, but also other GIS programs including MapInfo, Map Marker, SatScan, and QGIS, as well as statistical programs like SAS, SPSS and Tableau for research and surveillance projects.

Supported by funding from the Centers for Disease Control and Prevention (CDC), staff from the NYSDOH Environmental Public Health Tracking (EPHT) Program – also known as “Tracking” – has leveraged the free and open-source software program “R” to develop and maintain the Geographic Aggregation Tool (GAT).  R has grown primarily out of the statistics and data analysis space which is very popular and used extensively in public health research.

GAT is currently maintained by Abigail Stamm at NYSDOH who supports a GAT webpage on GitHub containing an extensive listing of documentation, developer tools, metadata, tutorials and more.  According to Stamm, EPHT staff like using R for a couple of reasons including:

  • R is free,  easily accessible to the public, and enables staff to share the GAT package with agencies that do not have an ArcGIS license or the training and resources to use any of the other geographical software. (Legacy versions of GAT, including a SAS version, are archived on GitHub.)
  • The GAT package automates everything.  This is particularly valuable especially if users include scripting options which bypass the Graphic User Interface (series of pop-up windows) and reduces likelihood of mistakes and makes recording and reproducing workflows and updating much easier.

GAT documentation and content on GitHub.

At its core, GAT aggregates, or dissolves, geographic areas (most commonly used is census tract geography) based on numeric values for each area, such as case or population numbers, as well as other demographic values such as median income.  Health researchers often want subject data at a higher granularity than the county-level, which can conceal or mask variation, especially in counties with a mix of urban and rural populations.  Also, showing data at town level won’t work because many rural towns have very small populations.  Areas with small populations are likely to have few cases, resulting in unstable rates and also putting confidentiality of cases at risk.

To overcome these limitations NYSDOH developed GAT to join neighboring geographic areas together until a user defined population and/or number of cases is reached to support the statistical analysis desired. This allows local health departments and others to use rates to identify hot spots for targeted interventions. GAT can also be used to produce maps at varying geographic resolutions required by the user.

How GAT Works

GAT requests user inputs through a series of dialogs, including menus, checkboxes, and text boxes, so no programming knowledge is necessary.  GAT reads in a polygon shapefile which must contain, at minimum, a character variable that uniquely identifies areas and a numeric variable to sum for aggregation.  A series of dialog boxes allows the user to select:

  1. A variable to uniquely identify areas
  2. One or two aggregation variables
  3. Optionally, a variable of areas within which merging will be preferred (ex. county)
  4. The value (sum) to which the selected aggregation variable(s) should be aggregated
  5. The preferred aggregation method: closest geographic or population-weighted centroid, least value, or ratio of two values

Depending on the specifics of the data and the type of analysis of interest to the user, GAT offers four types of aggregation methods:

  1. Closest geographic centroid
  2. Closest population – weighted centroid
  3. Neighbor with the lowest count
  4. Most similar neighbor

Applying different rules or criteria to the different GAT aggregation results produce contrasting results. These are samples of the GAT aggregation tool when applied to total population numbers in towns in Hamilton and Fulton (NY) Counties. Sample code for producing these maps can be accessed here.

GAT produces two shapefiles, an aggregated file and a crosswalk. GAT also produces a PDF of maps and a log of the entire process, including user settings, any warnings, and a brief data dictionary. (The PDF and log provide much more information than being shown in this article).   These files are designed to help evaluate and report aggregation results and standardize user process.  NYSDOH staff developed GAT to standardize and automate how to aggregate New York’s 4900 census tracts.

Using the different aggregation methods may affect the values of resulting aggregated areas in different ways. For example, it was found when testing GAT that in cases where a small corner of a census tract contained most of its population, aggregating by geographic versus population weighted centroid could provide very different results. Also, when developing aggregated areas for the Tracking portal, it was recognized the portal would be displaying disease rates.  NYSDOH staff felt the most suitable aggregation method for these population-based measures would be to aggregate to the closest population weighted centroid.  To check for areas with smaller populations and unusually large numbers of cases, users have the option to aggregate by case count rather than population (or in addition to population).

While access to health data and resolution of data varies across the NYSDOH (point/address, census tract, zip code, or municipal level), staff in the Tracking program receive hospitalization and emergency room visit data at point level (resident address) from the Statewide Planning and Research Cooperative System (SPARCS). Staff geocode these datasets using multiple programs including MapMarker, SAM in ArcGIS, and NYCGBAT and assign the encompassing census tract to the point data. Methods to assign to tract based on zip code or town and to impute tracts for records that cannot be geocoded due to incomplete address have also been developed.

 Other Applications of GAT

Public Use in New York State

 Working towards NYSDOH’s goal of making sub-county data more accessible, NYS EPHT is developing a platform which will display environmental health outcomes and exposures in an interactive mapping application.  The current EPHT data portal displays county-level indicators but EPHT staff are redesigning it to include subcounty data using the sub-county aggregated areas created by GAT.  This will assist local agencies with targeting interventions while maintaining privacy and confidentiality.

Tracking is also working with Health Data NY (HDNY) to display the sub-county data. The HDNY platform will serve as a data repository displaying data for environmental health outcomes including asthma, chronic obstructive pulmonary disease (COPD), and myocardial infarction (MI; heart attacks), since these are the health outcomes local agencies are most interested in when planning extreme weather-related mitigation and resource allocation. The data will be available for public download along with the shapefile and a brief description of how the shapefile was developed.

The CDC is also using GAT to develop sub-county areas for various health outcome indicators having piloted it for data from other EPHT grantee states for health outcomes including asthma, MI, and COPD.

Sampling Design

 GAT is also being used in the development and selection of sampling areas as part of Biomonitoring NY, a statewide biomonitoring project.  In the first year of sampling this effort has focused on the sampling of households on Long Island. To this end, the group aggregated 2010 census block groups within census tracts so that each aggregated area had at least 440 households. In the next step, remaining tracts that were too small to meet the household minimum were aggregated to neighboring tracts. After completing the final aggregation, study staff randomly selected 25 aggregated areas and mailed postcards and invitation packets soliciting participants in households in the block groups within the selected aggregated areas.  Read more about the project.

Biomonitoring is a way of measuring the amount of environmental chemicals found in the human body. It is an important part of New York statewide epidemiological research that seeks to determine levels of chemical exposure in the human body and help better understand whether chemical exposures are associated with health effects in humans.

Informing Policy

In 2019, NYS passed the nation-leading Climate Leadership and Community Protection Act (CLCPA; aka “Climate Act”) to empower New York residents to fight climate change at home, at work, and in their communities.  NYSDOH, along with several other state agencies, is providing support and assisting in the implementation of the Climate Act. One effort includes identifying disadvantaged communities (DAC) that could benefit from mitigation efforts and allocation of investments. GAT was used to create sub-county areas for several climate-impacted health indicators that will be analyzed with multiple other variables to define DACs.

Learn More About GAT

A variety of documentation is available on the use and development of GAT.  Slides (pdf) from a 2020 and National Association of Health Data Organizations (NAHDO) presentation by NYSDOH staff are available here including a YouTube video.  GAT slides and a video from the 2021 useR Conference are also available.   Most recently staff presented at the 2021 Place and Health Conference in November which included epidemiologists, health geographers, social and behavioral scientists, statisticians, data scientists, and public health professionals from all levels of government.

GAT poster that was presented at the 2021 Place and Health Conference (download here). A lightning talk is also available.

Contact:

Abigail Stamm, Research Scientist
Bureau of Environmental and Occupational Epidemiology
New York State Department of Health
abigail.stamm@health.ny.gov

GAT was written in R-2.9.2 under Windows XP and was revised and converted to a package in R-3.4.3 under Windows 10 using RStudio-1.4.1103 and devtools-2.3.2. The latest version of GAT was compiled in R-3.6.1 and runs in R-3.5.3 through R-4.1.1.

https://github.com/ajstamm/gatpkg

Cruising Empire State Data Portals for Geospatial Content

Posted on by
Reply

Expanding Suite of Online Portals Offer Data For All Types of GIS Users

Ever since the federal government launched Data.gov in May 2009 and President Barack Obama issuing an executive order in May 2013 establishing the Open Data Policy, as well as   Governor Andrew M. Cuomo signing Executive Order 95 (“Using Technology to Promote Transparency, Improve Government Performance and Enhance Citizen Engagement”) in March of the same year, governments at all levels across the country have responded in creating an array of online data portals with the intent of making data free and easily accessible.  Available and open to everyone including nonprofits, application developers, researchers and academia, business and industry, service industries, the geospatial community and everything in-between. Such portals purpose and intent is to promote government transparency, reduce data request and distribution costs, fuel business development and job growth, and drive innovation.  Open data portals are particularly useful in times of crisis or major incident when multiple agencies require access to the same data and/or may be limited in context of  data sharing agreements either ad hoc or formal.

Current Empire State Data Portal Offerings

Today, geospatial data portals across the Empire State have a variety of aliases including Open, Gateway, Hub, even the old school Clearinghouse title.  For the purpose of this article we’ll take a look at a range of the current publically available sites focusing on current content and some of the general functionality as it relates to the GIS/geospatial user. Sites included are those currently available in New York City, state government, and at the county and municipal.  It is by no means inclusive of all the sites across the state though for the ones itemized below, take a drive and see what you find.

 Map Services vs. Files

Accessing, or consuming, geospatial content has changed significantly since the early years of data sharing.  The term “open data” was barely in the geospatial vernacular in old school GIS with many sites providing access to data which was governed by formal data sharing agreements and/or requiring some kind of organizational membership.  Offering file-based downloads to support, which at the time, was a heavy-client software-based world of GIS computing.

While file-based downloads from the myriad of portals across the Empire State are more than adequate for most users of the sites, GIS/geospatial users often have software enhanced capabilities to consume large volumes of data content in both client and browser applications via map services.   Consuming web services is now even an option within certain versions of AutoCAD which greatly expands the use of government authored and published geospatial content.

Given the option of having to take the time to download numerous individual files vs. consuming one map service with many data layers and picking only the data layers needed – the choice is often pretty obvious.  As such, data portals which offer geospatial content as a service (or by extension developer tools to provide similar functionality) increasingly have added value to the GIS/geospatial user.

New York State Government

HEALTH.DATA.NY.GOV & DATA.NY.GOV

Launched approximately around the same time in 2013,  these two portals provide lots of file based content for the GIS/geospatial community.  Built on top of Socrata software (Tyler Technologies) the two sites differ only in content. DATA.NY.GOV serving more as a generalized repository for state agencies (and selected local governments) and HEALTH.DATA.NY.GOV focusing specifically on public health related datasets.  Creating an user account (if needed) provides developers and publishers to identical tools and documentation to each site including customization options, database connections and API endpoints, metadata documentation, visualization tools, and more..  Both portals as part of the broader Open NY initiative.

While centralized data access and data visualization (charts, dashboards, etc) have always been a primary Socrata focus, the software also includes a basic toolkit  to map and render datasets which include coordinate geometry.  Both sites also include the ability to download datasets in a variety of formats including KML, KMZ, Shapefile, and GeoJSON which can be used by a wide range of GIS software platforms.

Users of DATA.NY.GOV site need to be aware this site ALSO includes content from other statewide Socrata sites (HEALTH.DATA.NY.GOV, NYC Open Data, and  Open Data Buffalo) so it is important to use the filtering function in the lower left corner of the page after the initial data theme/category has been selected.  For example, in the image below, after selecting the “Transportation” category, the application returns 521 records – which includes those meeting the same criteria from the other three statewide Socrata sites. (And in this category many from the NYC Socrata library.)   After filtering for “This site only” (DATA.NY.GOV) the application returns just 251 records.  A small nuance but filtering does help if the search is specific to certain jurisdictional or geographic footprints.  Also, when searching for state agency data across the available statewide portals, users will begin to see that some agency data is published and available on more than one portal.  For example, some state agency data can be found on  DATA.NY.GOV, CUGIR, and the state Clearinghouse.

The filtering function on DATA.NY.GOV limits the search to specific domains

Plenty of geospatial data in both of these state portals albeit as the GIS community has come to know during the COVID-19 pandemic, address or large scale data is hard come by in the public health space.  Respectfully a whole other discussion and until there are fundamental changes in how public health data is published, it will be difficult for non-public health agencies, advocacy groups, or the general public to access such data at units of geography smaller than the county level.  Or sometimes if one is lucky, finding some at the zip code level.

For additional information on how to export Socrata data to other formats and/or platforms see this help document.  Also, Koop software is an Open Geospatial ETL Engine which enables users to leave geospatial data where it lives and transform it into a variety of formats including GeoJSON, CSV, KML, a Shapefile, or Feature Service dynamically.

There is apparently ongoing work/research within the state GIS offices with regard to both mapping NY Open data using the Socrata JavaScript API as well as enabling the Socrata platform to use the NYS GIS Program Office geocoder.  If successful, such efforts will greatly expand the Socrata data portals for the geospatial community.

NY Department of State Geographic Information Gateway

Launched in 2015, the Gateway is administered out of the NY Department of State, Office of Planning, Development and Community Infrastructure and is now home to over 700 datasets. The Gateway is an online mapping application that makes it possible to view geographic information in its native viewer or Google Earth and users can also download information in multiple file formats or even pull into other online mapping applications.  While the original geographic footprint of the Gateway focused on New York’s offshore planning area in the ocean and State’s portion of the Great Lakes, the repository now contains a wide range of geospatial datasets covering the entire State.

The Gateway has datasets  grouped  thematically and made available as map services. All content includes accompanying metadata enabling users to identify which data layers to select and easily consume from within the composite map service.    Perfect for desktop or web client mash-ups and analysis.  The site also includes links, integration, and content with crowdsourced apps.

The Gateway is a must see site having itself apart from other NYS departments in context of building a geospatial data portal with so much functionality and content.  There is a good chance you’ll find content for your GIS project here.

The Gateway data browser has icons for actions that can be taken with desired datasets; download, view in native map viewer, export to Google Earth, information for ingesting as a map service and metadata.

 NYS GIS Clearinghouse

The Clearinghouse was originally implemented in the late 1990s to facilitate statewide data sharing.   For many years it was the defacto #1 portal for all things/files GIS.  The focus being based on formal/written data sharing agreements between government and non-profit organizations. Amazingly in 2021 these agreements are still part of the Clearinghouse administrative structure though it does not get in the way of non-data sharing members – even those from industry and business – from downloading many datasets.  Give credit to those individuals who helped create the original Clearinghouse as it was one of the first of its kind at the time.

Much has changed since the Clearinghouse was initially launched which has impacted the data component of the site.  Time itself, and in this case decades, has proven to be a major challenge.  Publishers have come and gone, numerous retirements and point of contacts no longer exist, and limited metadata and datasets being out-of-date.  It’s a next to impossible to manage and keep most of the data files up-to-date and uniquely different in portals like the Clearinghouse.  Given the new generation of geospatial data portals that  we should expect to continue to emerge (particularly at the local level), the relevancy gap of the Clearinghouse’s data file inventory will likely continue to widen.  To say the least of the role of other similar data and web services available online via other free government and business sites such as The National Map, EPA Envirofacts, and ESRI’s Living Atlas.

The upside is the Clearinghouse is emphasizing and pushing data content  as  web services having placed this “option” at the top of the data catalog list. Seemingly to encourage users to “try this first” before sitting down and mining through the catalog on a file-by-file basis.  Much of the web service content are from the go-to State GIS office program areas (imagery, elevation, addresses, geocoding services, and to a limited degree tax parcels) albeit other state agencies are contributing services as well.   Kudos to the state agencies are keeping their individual file catalogs up-to-date and current albeit as earlier mentioned some of these state agencies are placing data on multiple portals requiring users to search in more than one location.

Instead of performing file based searches, users can access content via map services from a variety of state agencies

Hopefully, the legal shackles of the Clearinghouse data sharing agreement – instituted in the previous century – alone do not continue to influence the long term focus of the Clearinghouse with regard to managing and publishing individual datasets. One day, the statewide geospatial community would probably be better served when all of the remaining relevant  Clearinghouse datasets are consolidated into DATA.NY.US or CUGIR.     And perhaps leave the Clearinghouse to focus just on publishing data as a service with the current capacity it has built.

Municipal Government

NYC Open Data

Another Socrata site, NYC Open Data is the big kid in the statewide open data sandbox with over 3,000 datasets grouped into just five categories available from nearly 100 reporting NYC departments.  In addition to just datasets, the site also includes files, documents, and external links.  So, if you are looking for geospatial data for the NYC geographic footprint this is the place to go.  The site has been around for several years already and is the creation of the  Mayor’s Office of Data Analytics (MODA) and the Department of Information Technology and Telecommunications (DoITT). As a hub of analytics in the City, MODA advocates for the use of Open Data in citywide data analytics and in the community.

The NYC Open Data Portal also provides excellent narratives and examples of how geospatial data is used and is critical to the operations of city government.

NYC Open Data exemplifies the use of open data giving rise to a broad user community including organizations such as BetaNYC which hosts events such as the NYC School of Data, Open Data Journeys, various Open Data Classes, and in concert with the Mayor’s Office, the signature NYC Open Data Week – this year March 6 – 14. And data from the portal is routinely used in the NYC BigApps competition.

To fully appreciate how far open data has come in NYC, take a look at the list of publications and reports on this web page which chronicles its growth over the past several years.  And if you need further convincing, take a look at the NYC Open Data Project Gallery.

Go to the site and drive around.   Its worth the ride.

City of Rochester (DataROC)

Cities are great units of geography to build data portals in that their governments often oversee management of all the geographic features and data within the municipal boundary footprint.  Albeit this does vary on occasion with regard to utilities and some infrastructure  Geospatial data is consistent at the city level as government business applications are built and deployed to capture data to the same geographic extent.  As city applications continue to be geo-enabled, and open data initiatives are initiated,  this geofencing of the data provides a consistent geographic extent and a great framework to support a geospatial data portal.

DataROC was launched in 2020 and due to the city’s long time relationship with ESRI and staff who are statisticians and/or business intelligence analysts.   “ESRI’s open data framework was a good fit for the city” according to Kate Day, Chief Performance Officer, Mayor’s Office of Data Analytics & Performance.   Prior to the rollout of DataROC, the city already had both a police data portal and a map gallery of different ESRI tools, so the first major part of DataROC was to bring previously decentralized content together and make it easy to search. GIS staff in the city’s IT department assisted as change champions to make sure that the wider GIS community spread out across the City were aware of what the focus of the portal was to be and drive more traffic to the  work analysts in various department had done over the years.  Currently there are 203 items in the city’s open data catalog containing data files, apps, maps, documents.  If you need geospatial data for a project inside the Rochester footprint, this is the portal to visit and search for data.

DataROC offers easy access to data, applications that feature maps curated for a specific purpose, featuring up-to-date data, and a gallery of web maps to get around the city

With regard to an “example” as how specific governments vet data to their portals, the City of Rochester “publishes anything that is classified as public data / documents / applications that have been requested to be published by an internal or external stakeholder, but only after they have been approved by a Data Governance Committee”.  This committee ensures each dataset is truly for public consumption (essentially that it poses no individual or group harm for being published, that it would not violate any law/regulation to publish, and that the dataset meets quality standards to accurately represent whatever subject is being depicted or summarized).

Other illustrative statewide local government data portals include the City of Buffalo (Open Data Buffalo), City of Albany (openAlbany),  Westchester County (GeoHub), and City of Syracuse (DataCuse).

Academic

Cornell University Geospatial Information Repository (CUGIR)

I’ve always been a huge fan of the CUGIR portal and related program offerings.  Housed in academia and on the fringe of the statewide GIS mainstream, its original homegrown portal was built  and developed in Java in the 1990s.   The current CUGIR portal was launched in January 2018 and  is is built on several modern open-source components.  The front-end interface uses GeoBlacklight — https://geoblacklight.org/  which is a collaborative project being developed by several universities, including NYU, Minnesota, Stanford, Princeton, UC Berkeley, Cornell, and others.  CUGIR has made various customizations, some of which have been incorporated back to the project.

Subjects such as landforms and topography, soils, hydrology, environmental hazards, agricultural activities, wildlife and natural resource management are included  in the CUGIR catalog. All data files are cataloged in accordance with FGDC standards and made available in widely used geospatial data formats.

Behind the scenes, CUGIR uses Solr for searching and faceting (also known as “data filters”).  The site  use GeoServer to access copies of the data in a PostGIS database to generate map previews via publically-accessible WMS/WFS web services for each dataset.  Amazon S3 is used to host all the downloadable data as pre-generated zipfiles, so downloads are quick and reliable. The site uses  Leaflet to display the map on top of a CARTO basemap (which is built from OpenStreetMap data the web viewer.   Maintaining and supporting the site is long-time CUGIR GIS front man, Keith Jenkins.

Today, CUGIR’s datasets total about 15GB.  The total number of “datasets” listed in the catalog is currently 458, which appears much smaller than the nearly 8000 it used to be because in the old system CUGIR had content split up into many smaller downloads to accommodate the bandwidth limitations of the late 1990s and early 2000s.  So to leverage newer technologies and internet speeds, several data series were merged into statewide datasets.  As part of the process, CUGIR also converted old formats like ArcInfo Interchange (.e00) files to shapefiles wherever possible.  As part of the site upgrade, old links which had bookmarked or published were updated as well.  Today, NYS DEC and USDA NRCS have the most CUGIR records with other large catalogs from NYS Department of Agriculture and Markets and Cornell Institute for Resource Information Sciences (IRIS).   Data is also available from nearby Tompkins County GIS.   Excellent filtering tools enable users to search data by  several different categories  and download datasets in Shapefile, PDF, or KML format and export to either KMZ or GeoJSON.

CUGIR focuses exclusively on geospatial data which sets it apart from other general sites like DATA.NY.GOV.  The NYS GIS Clearinghouse and the NYS Geographic Information Gateway are more similar to CUGIR with each having their own unique datasets which in some instances can be confusing for someone looking for data and not sure where to look.  Additional CUGIR functionality includes pointing to and maintaining an index map to external datasets such as the National Elevation Dataset (NED).    Jenkins and his CUGIR team hope to build out similar functionality and integration with other Empire State geospatial portals in the future.

 

The Geography of Homelessness

Posted on by
Reply

Background

The Institute for Children, Poverty, and Homelessness (ICPH) is a New York City-based policy research organization focused on family homelessness in New York City and throughout the United States.

Formerly known as the Institute for Children and Poverty, ICPH was founded in 1990 as federal and state governments looked for ways to address growing rates of family homelessness. Through the examination of empirical, quantifiable data, ICPH seeks to inform and enhance public policy related to homeless families, with an emphasis on the impact on children.  The New York City numbers are staggering with over 12,000 families—including 25,000 children—calling a city shelter their home.  A small city in itself.

And unfortunately, the experience of housing instability and homelessness in New York City’s neighborhoods goes far beyond what shelter numbers alone show. In school year (SY) 2015–16, City schools identified four times as many homeless students as there were children living in shelter, and the current structure of the City’s shelter system is ill-equipped to meet the long-term stability needs of homeless children and families living both in and outside of the City’s family shelters.

ICPH examines the demographics of this growing population, the challenges these families face in becoming self-sufficient, and the programs that are most effective in helping them transition out of homelessness in their reports, research-based books, and policy research commentary.   ICPH’s publications inform government officials, policymakers, other research organizations, advocates, academics, and service providers from across the metropolitan region to promote a robust, evidence-based dialogue.  Central to ICPH’s core work is the intersection of statistical analysis, data visualization, and mapping technologies resulting in a wide range of geospatial-related products and applications.

ICPH Geospatial

ICPH utilizes several mapping, desktop publishing, and statistical software packages in support of publishing online and hardcopy products.  While ICPH actively promotes the use of traditional interactive maps, the increasing use of hybrid products i.e.  maps/charts/tables/graphics (a.k.a. Inforgraphics) are widely used as part of delivering the organization’s message.   Inforgraphics are increasingly being used across the GIS community to provide graphic visual representations of information, data or knowledge intended to present information quickly and clearly.

Managing geospatial applications and development at ICPH is Bronx-native Kristen MacFarlane.  She arrived at ICPH in December 2016 after a stop at the New York Botanical Gardens.   She received her undergraduate degree in Natural Resources from Cornell University and her masters in Geographic Information Science (GISc) from CUNY Lehman College in the Bronx.  Kristen oversees an organizational geospatial tool box which includes a several software packages including Tableau and ESRI.  While most ICPH projects are NYC focused, the reach of the organization’s research and work covers other areas of the country as well.  Projects are funded by foundations, grants, and other contributions and initiated by ICPH professional staff and industry colleagues.  MacFarlane notes that data used in the studies is mostly gathered from open data portals or through Freedom of Information Law (FOIL) requests.  As a primer on her involvement with ICPH, see her November 17, 2017 GIS Day blog post GIS Day: Discovering Family Homelessness Through GIS.

ICPH Geospatial and Data Visualization Applications

Suspension Hubs Interactive Map

In New York City, there are 102 Suspension Hub schools serving nearly 3,500 homeless middle schoolers. These schools suspended more than 6.6% of their students overall in SY 2015–16—nearly three times the city’s rate of 2.5%—and their rate of suspension among homeless middle school students actually increased during the years since SY 2010–11. Understanding where Suspension Hubs are located represents an opportunity to offer these schools more support and resources so that they, too, can turn the tide like the 83% of other schools with declining suspension rates. The Suspension Hubs Interactive Map was developed in Tableau using data generated and analyzed with Stata.  Project data is from NYC Department of Education for SY 2015-2016.

In New York City, there are 102 suspension hub middle schools where students are disciplined at extremely high rates.

Interactive Map and Atlas of Student Homelessness

Unless current trends change, one in seven New York City public school students will be homeless during elementary school. More than 140,000 children attending New York City public schools have experienced homelessness within the past six years.  Visualizing the magnitude of this issue is the updated ICPH application New York City Interactive Map of Student Homelessness as well as On the Map:  The Atlas of Student Homelessness in New York City 2018.  Both products provide a wide range of  data to allow people from different fields to tailor and engage with data on student homelessness in a way that is meaningful to them and the unique needs of their organizations and locales in the city. The Atlas documentation is from Section 1 only of the report which is available for download from this link.  The August 2018 update is more similar to ones that ICPH presents as static publications.  The full report contains maps generated with the ArcGIS client which are exported for inclusion in online and hardcopy publications.  On a larger scale, ICPH created a similar interactive nationwide map – The United States of Homelessness – scale using data from the U.S. Department of Education.

This interactive map provides users with the ability to see what student homelessness looks like in every NYC neighborhood.

Neighborhood Look at Domestic Violence as a Driver of Homelessness in NYC

Family homelessness has many different causes and drivers. One of the more common drivers in NYC are families being pushed into homelessness because of experience with domestic violence. This map demonstrates the rates at which domestic violence is driving family homelessness in neighborhoods across NYC.

Percent of families with children eligible for shelter due to domestic violence by Community District July 2014 – December 2015. Data source: NYC Department of Homeless Services.

Food Insecurity & NYC’s Homeless Children

A household is considered food insecure if there is a lack of access to adequate, healthy food for all household members.    In NYC, students are eligible for free or reduced-price lunches  if their family income is 185% of the Federal Poverty Level or less. Other programs such as the federal Supplemental Nutrition Assistance Program (SNAP) provides food stamps or SNAP benefits to households with incomes 130% of the Federal Poverty Level or less, in addition to employment requirements.  This ICPH interactive map enables users to access localized data on school lunch programs, food insecurity by community district and SNAP benefits.  An excellent User Manual is provided as well.

Data sources for this interactive map include NYC Dept. of Education, Food Bank for NYC, Feeding American, Data2Go.nyc, and U.S. Census Bureau, 2012-2016 American Community Survey estimates.

Summary

While the core focus of ICPH’s research and advocacy continues to be in the New York City region, its work covers the entire United States.  Detailed homelessness related studies have been done in specific metropolitan areas.  Similar the nationwide scope of the United States of Homelessness study referenced above, ICPH released the Student Homelessness and Food Deserts interactive map during Hunger and Homelessness Awareness Week, November 12-16, 2018. Other student-focused homelessness studies have been conducted in California, Georgia, Seattle, and New Jersey.  A wide range of publications and reports compiled by ICPH staff and collaborating organizations focusing on homelessness-related topics such as education, health and well being, housing, and community issues can be accessed  by clicking on the image below and then using the Research tab to browse Reports and Interactive Data.

“Geospatial tools are essential for our organization as we work to give the public a clearer, more informed picture of what child and family homelessness looks like in their community and across the country,” said Kristen MacFarlane, Senior GIS Analyst at ICPH. “Homelessness is a national crisis but a local issue and the specific ways that homelessness varies between localities and even neighborhoods are important to understand in order to develop policy solutions for those experiencing it. GIS software helps ICPH convey the complexities and the scope of this growing crisis.”

Contact:

Kristen MacFarlane
Senior GIS Analyst
Institute for Children, Poverty, and Homelessness
36 Cooper Square, 2nd Floor
New York, New York  10003
KMacFarlane@icphusa.org

Watershed Agricultural Council: Geospatial in Upstate NYC Watershed Lands

Posted on by
Reply

The New York City Watershed is the largest unfiltered surface water supply system in the United States. Every day this region in upstate New York’s Catskill Mountains provides more than a billion gallons of clean drinking water to the more than nine million people residing in New York City and upstate counties. The system includes 19 reservoirs, three controlled lakes, and a watershed that spans more than a million acres. In some cases, water travels more than a hundred miles to reach New York City.  While other large US cities typically own the land that drains into their reservoirs – New York City’s lands are unique:  Nearly 75 percent of the watershed lands are privately owned with NYC and the State of New York owning a mixture of the remaining lands.  Consequently, New York’s water quality depends in large part on the actions of the 30,000 landowners who live, work, and recreate on watershed lands.  Overall, nearly 70% of the watersheds are forested with the remaining lands in predominately agricultural use.

The Watershed Agricultural Council (WAC) is one of two primary watershed partners (Catskill Watershed Corporation being the other) locally-based not-for-profit corporations specifically created to assist New York City Department of Environmental Protection (NYCDEP) with the administration and implementation of watershed protection and economic development programs in its water supply lands. The WAC is involved primarily with natural resource and land conservation programs, whereas the Catskill Watershed Corporation primarily oversees community infrastructure and economic development programs. Many other watershed partners serve in various capacities, such as oversight, advice, public education, and communication.

WAC Geospatial

WAC GIS and geospatial applications are built to support four major program areas within the NYC watershed lands:  Forestry, Agriculture, Conservation Easements, and Economic Viability.    Managing these applications and infrastructure in their three offices [(Walton,  Hamden , and Yorktown (East of Hudson)]  is the responsibility of Nadine Trahan who joined WAC three years ago after having attended both undergraduate and graduate school in New Zealand.  Nadine developed and maintains a ESRI Geodatabase to support enterprise geospatial applications.  She notes wireless connectivity and high speed internet connectivity can be a “challenge” in the rural Catskills which plays into the deployment of the enterprise applications.  Given the connectivity issues, Nadine uses ArcGIS Portal to design and download applications to mobile devices for in-field deployment:  Data collected remotely is synchronized with the enterprise geodatabase when field devices are connected to the network upon return to the office.  She currently supports approximately 40 users involving everything from high-end client software apps to browser-based applications for the more casual user.  This later category of user being staff which use an internal web map to edit point locations of Pure Catskills members as part of the Economic Viability focus.
Continue reading →

Geospatial Business Spotlight: CycloMedia Technology, Inc.

Posted on by
Reply

Company Name:       CycloMedia Technology Inc.

Location:                   Berkeley, CA

Website:                    www.cyclomedia.com

Employees:               120 Globally

Established:              1981

A privately held company, founded in the Netherlands over thirty years ago, CycloMedia Technology, Inc. is a rapidly expanding international company with staff in Syracuse, New York.  Annually, CycloMedia captures and processes approximately 160,000 linear miles of 360° panoramic street level images, representing over 48 million recordings per year. The company holds five key image capture technology patents and specializes in the local government market. Currently the company has projects with the cities of New York, Philadelphia, Washington DC, as well as  Franklin County, OH (Columbus), Dakota County, MN (Minneapolis) and Jefferson County, KY (Louisville) to name a few.

Specific benefits of Cyclomedia technology include:

  • GeoCyclorama images and basemaps are geographically linked
  • Provides ability to conduct searches with address, postal code or coordinates
  • Integrate with interprise applications and work processes using the GlobeSpotter API
  • Determine the location and dimensions of objects in GeoCyclorama imagery
  • Ability to create precise point, line and area measurements
  • Overlay enterprise spatial data on the GeoCyclorama images
  • Supports Open Geospatial Consortium (OGC) standards
  • Integrate data in formats such as WFS, WMS, Shapefile and GML
  • Create reports and bookmark locations of interest
  • Users can save and utilize GeoCyclorama images for reporting and analysis functions.

Geospatial Products and Services:

A.  GlobeSpotter

GlobeSpotter (web) is an interactive web-based application that provides cloud access to GeoCyclorama imagery and tools for measuring, collecting data, and reporting.  The software and data are directly available from CycloMedia’s software as a service (SaaS) model.

GlobeSpotter (web) screenshot with area measurement and reporting window 

GlobeSpotter for ArcGIS Desktop  provides visualization, measurement and overlay tools within the ArcMap or ArcGIS Pro allowing a user to fully leverage the power of GeoCycloramas within the ArcGIS client environment.  Specific capabilities include:

    • Open GeoCyclorama imagery from a recording point layer added to the map
    • View one or multiple GeoCyclorama images
    • Measure on GeoCyclorama images
    • Use the native editing capabilities of ArcGIS to collect features in 2D or 3D
    • Ability to save and utilize GeoCyclorama images for reporting and analysis functions.

Continue reading →