Tag Archives: open data

March 2023: Tax Parcel Data Sharing in New York State

Posted on by
2

Half of New York’s Counties now make parcel data available for free through Open Data efforts though barriers in specific governments still need to be overcome before statewide data is available

Not too long ago, I was communicating with a colleague about how some governments still, in 2023, continue to charge or license tax parcel data.  At first, somewhat amazing considering how far the Open Data movement has come over the past 10-15 years across the Empire State.  As recently as late 2022, information from the NYS GIS Program Office, which monitors the availability and sharing of tax parcel data statewide, shows that nearly 50 percent of NYS counties are making tax parcel data available for free through various means.  Great strides being made perhaps suggesting that one day the barriers prohibiting the sharing of parcel data across the entire will finally be nonexistent?   And that the gripes and complaints I was sharing in conversation with my colleague would go away.

Or maybe not.

Since November 2022, this map has been updated and NYS GPO now notes that half of the statewide counties currently share tax parcel data

Issue Was Discussed Early On – What Happened?

In the early days of statewide GIS development (circa 1990s) the GIS community was fortunate to have Bob Freeman, former Executive Director of the New York State Commitee on Open Government (COOG) participating in the growth of the GIS movement.    Bob seemingly recognizing and witnessing the formation of a technology which would have a tremendous impact on government records and data.  Almost knowing that as part of the new technology, digital versions of data would become just as important or even more valuable than hardcopy format.  He was involved in statewide GIS conferences and contributed in many ways.    All said and done, he was a staunch advocate of making GIS data available – largely long before the “Open Data” concept was framed.

I was only able to find a few of his opinions searching the COOG archives and ultimately reached out to the current COOG Executive Director Shoshana Bewlay to both bring to her attention of governments still charging for tax parcel data as well as to see if she, or her staff, might be interested in revisiting the issue.  I heard back promptly from  Kristin O’Neill, Assistant Director who was kind enough to send me a link to all of the COOG opinions (7) which included “GIS”.  The links to all of the GIS-related opinions are below which can either be read through in its entirety or skimmed-over.   There is a lot here some of which is better interpreted by an attorney.  7507, 11230, 13575, 14366, 15058, 15695, 19246.  Looking back, I’m not sure why I thought the issue had gone further along in the legal system and had been resolved to a more definitive degree. But these few opinions suggest otherwise.  

More than once, COOG responded to the question/issue of an individual or organization questioning the basis of a government organization either selling or licensing tax parcel data.  And in general, each time, the COOG response – or opinion – was the same noting that electronic records (digital) records are no different than hardcopy records and should be made available at the cost of duplication.  (Keeping in mind original Freedom of Information Law (FOIL) costs of duplication were originally based on hardcopy records (i.e., for example 25-cents per page) and did not/does not have specific rates for digital data duplication.  Today, staff time, tapes, etc., can be included in the charge. However, since COOG opinions are advisory in nature only in context of FOIL, the early inquiries did not result in any formal legal challenge being launched on any level against any entity with regard to the selling of tax parcel data.   Suffolk County early on became involved in litigation on licensing/copyright issues (referenced in one of the opinions) which is still in place today.  But by the late 1990s, the selling of tax parcel data was in place and had become business as usual.  It seems the geospatial community is no further along in “the challenge” to the selling or licensing of tax parcel data than we were some thirty years ago.

Continue reading →

Envisioning the Future of Buffalo’s East Side

Posted on by
Reply

AGOL viewer, data HUB, and other geospatial tools used in helping develop WITHIN East Side Plan

Community focused GIS projects are always a favorite of mine to write about.  Highlighting how geospatial tools can be used by community groups – including nonprofits which are often project sponsors – to better help visualize and understand the vast array of environmental, regulatory, business and public health, and cultural data which impacts their daily lives.  One such project located in Buffalo’s east side was brought to my attention in a recent communication with Lisa Matthies-Wiza, Director of Geographic Information Services at Erie County.  

WITHIN East Side

The WITHIN East Side project is one of many projects of LISC Western New York (WNY) and its larger parent organization LISC New York.  program.   WITHIN East Side  focuses on simplifying the neighborhood planning and community development process, and ensures development is driven by resident leaders and community groups.  As an open, inclusive, community-driven initiative, WITHIN East Side brings together neighborhood voices, trusted partners, and supportive funders in an effort to better the wellbeing of residents using their own visions. Together with LISC NY, the WITHIN East Side stakeholders collaboratively identify projects that sustain the positive momentum happening in local neighborhoods.

In early 2020, LISC NY began planning with community partners in three geographic focus areas in the East Side under the WITHIN East Side program.  The East Side of Buffalo, which is the heart of Buffalo’s Black community was chosen as it is purposeful to build upon and lift up the long-time visions of residents and community leaders in neighborhoods most impacted by historic disinvestment, environmental, structural, and systemic racism.  The additional impact of the racist mass shooting on May 14, 2022, that killed 10 people, at a local supermarket within the planning area, furthered the importance of amplifying the community’s voice and vision for the future.

The project was designed as a two-pronged approach–integrating economic development and quality-of-life planning.  Closely intertwined, both  economic development and quality-of-life planning require understanding community history and dynamics, collaboratively identifying projects, building relationships with diverse stakeholders, and turning community priorities into progress.

The WITHIN East Side project was broken into three distinct study areas – each of which was studied in more detail for a wide range of demographic, cultural and economic development issues

Background

LISC NY contracted with Prospect Hill Consulting (PHC), a local minority and women owned consulting firm which had responded to a formal Request for Proposals (RFP) which was issued in January 2021.  GIS mapping and analysis was a requirement and the LISC NY team knew GIS services were important to supporting the overall East Side project.  PHC was selected to perform elements A (GIS Mapping [including an online mapping tool request and a data HUB) and B (Scenario Planning).   While LISC NY had a vision of what the final plan might look like,  PHC was instrumental in assisting LISC NY in developing maps and visualizations used in the final plan and in community engagement sessions throughout the process.

Existing 2021 land use is just one of dozens of data layers in the WITHIN East Side AGOL viewer. The rich database consists of local, regional, state and federal datasets.

Leading geospatial development of the East Side project from PHC was Jenny Magovero, President and Co-founder. Ms. Mogavero has been working in the GIS field for 23 years covering geospatial applications in community planning and environmental sciences as well as work in data visualization and  project management.  Mogavero created all of the maps in the East Side plan using the ArcGIS client as well as for spatial analysis and modeling.  The project web map is ArcGIS Online based with the data stored in a Hub Site (to allow for data sharing) and was designed about halfway through the GIS mapping and analysis task to support the WITHIN planning process itself.

One of the many excellent graphics iWITHIN East Side report. The map on the right identities areas within the study area where supermarkets are located – a significant issue for residents with limited transportation options.

LISC NY wanted to be as transparent as possible and allow stakeholders (i.e. the community/public, organizations, etc.) to work with the data and download and use it if they needed.  To this end, the AGOL interactive map was used in community outreach and meetings early in the project and as new data came online (i.e., stakeholders requested to see additional themes) PHC continued to update the AGOL viewer. PHC designed the AGOL viewer to serve as a communication medium that allowed LISC NY to show project progress, survey the community on what themes needed to be covered and/or identify gaps, and to act as a platform to continue engagement with the community in the future. 

Economic Development issues are paramount in the study such as the role of small businesses, inflow of non-residents that work – but do not live – in the area, as well as the concentration(s) and accessibility of employment opportunities.

Data collection and development was specific to the mapping and analysis that the LISC NY team needed for the WITHIN plan itself.  As LISC NY has significant business relationships within the greater Buffalo community, they were able to collect a lot of data from local, state and NGOs. In addition, PHC collected publicly available data from the City of Buffalo, Erie County (parcels, land use, etc.), New York State  (NYSDEC, NYSDOT, etc.) and federal datasets (EPA, Census, etc.).  Additionally, PHC developed data from reports or datasets that were anecdotally described by stakeholders (like the air quality buffer, buried portions of the Scajaquada Creek, key intersections, etc.).  Project data was also obtained from utilities, Google, OpenStreetMaps, and ESRI.  

Most of the project data is made available through the LISC WNY Open Data Hub which was also designed and continues to be supported by PHC.  (Some providers did not want their data to be shared so it is only available for viewing in the application.)   

No AGOL account is necessary to access the HUB and tags help users to easily jump to specific themes of data presented in the final plan for download.

PHC used a bevy of GIS,  statistical packages, and desktop publishing software tools to generate the maps and graphics in the report including: 

  1. ArcGIS Online to collect stats for the neighborhood that were ACS 5yr census based (https://doc.arcgis.com/en/arcgis-online/analyze/enrich-layer.htm)
  2. Geospatial processing tools to buffer, clip, summarize, and run overlay analyses
  3. Spatial Analyst tools to develop walkshed layers (from points provided by Walkscore)
  4. Tableau Desktop and Excel to conduct data exploration to emphasize trends (or data insights) with graphs/visualization on  map layouts
  5. Adobe InDesign for page layouts and other graphics to be consistent with the LISC NY brand book (design guide) including colors, fonts, and logos.
  6. ESRI’s Hexagon sampling tool to visualize parcel data classes (i.e. land use, vacant parcel density, ownership) at the scale of the plan areas. 

Residential ownership type is an important variable in urban studies. Data presented in the figure above suggests there continue to be many opportunities in increasing owner occupied properties in the study area

Summary

While consulting architecture and engineering companies continue to expand their geospatial offerings to governments and regional programs in 2022, particularly in the infrastructure and public works space, planning firms such as PHC serve in a unique space focusing on efforts which include applying geospatial tools as part of important community-based projects which often include significant public input and involvement.  Tools such as AGOL provide a great  framework in which to engage the public with regard to adding and removing data content and visualizing potential scenario outcomes.  A particularly useful tool given the enormity of the data used in the WITHIN East Side plan.

Reflecting on PHC’s involvement in the study, Jenny Mogavero notes:

“LISC NY is an integral part of, and key economic engine within the Western New York region.  Prospect Hill was honored to partner with the LISC NY to use GIS data, visualization tools and spatial analysis methods to not only present the existing conditions of our city’s East Side neighborhoods, but also reveal the deeper insights that occur when we overlap data-driven stories across multiple thematic lanes.  We were thrilled that our work supported the teams’ consensus building goals to develop a vision of a prosperous future for an important part of our City.”

Tyra Johnson Hux, WNY Director of Operations, also reflects on PHC’s work and the use of geospatial tools in the project adding:

“WITHIN East Side amplifies the visions of residents and community leaders in neighborhoods impacted by historic disinvestment, environmental, structural, & systemic racism. Showing their stories through, not only their own words, but also data was critical to helping our stakeholders generate a bold, authentic and comprehensive vision with an emphasis on implementation.  PHC supported the WITHIN East Side Plan by working with the project team to identify, collect and normalize relevant GIS data from neighborhood, government, academic and other partners. Additionally, they worked with us to analyze and visualize the community experience through maps and infographics.  The online interactive map and data warehouse they developed made it possible to share data in a way that breaks down silos, facilitate  strategic planning and continue community conversations.”

Contact

Jenny Mogavero, GISP
Prospect Hill Consulting LLC | Principal
716.432.9053 | www.prospecthill.co
jmogavero@prospecthill.co

Lashay Young, Director
External Affairs for LISC NY
https://www.lisc.org/
LYoung@lisc.org

Empire State GIS/Mapping DIYer Phenom: Andy Arthur

Posted on by
Reply

Self-taught hobbyist has a treasure chest of geospatial content on website

One of the benefits of writing about all-things geospatial in Empire State is sometimes I just don’t know what I’ll come across.  Looking for this thing and finding that.   Starting in earnest on an article about a certain GIS channel and a couple days later finding myself having completely jumped the rails and find myself way over there writing about Channel Z.  (Yup, that static in the attic).  Or ending up on a cool or fun website not really knowing how I got there.

Case in point:  Interactive Maps by Andy Arthur.  Empire State mapping DIYer extraordinaire.  Just a hobby.

When I first stumbled onto the site and having spent some time driving around, I realized I needed to find out who was behind it all.  It definitely isn’t the kind and feel of the traditional geospatial website I normally include or reference in my blog, but enough interesting – and yes, quite different – content to dig a little deeper.  And glad I did.  This is not a blog post to focus on a particular topic or concept, but rather just more of a pointer to the URL and let you take away from the website what you want.  

It turns out the person behind all of this is Andy Arthur, who by day, is Deputy Director of Research Services in the NYS Assembly.   “I have no formal GIS training, as things were still pretty primitive back when I was in college (SUNY Plattsburgh)  in the early 2000s especially when it came to web services, online data and open source software” says Arthur, “computers were a lot less powerful back then. I remember vaguely hearing a bit about Remote Sensing when I was involved in the Environmental Science Club in college, but it wasn’t something I ever used.“

Since then and working on his own, Arthur picked up QGIS (and the accompanying PyQGIS developer tools), as he was looking for a way to make his own topographic maps because he wasn’t happy with what was available on  the Internet. He later found out he could FOIL a primative campsite shapefile from NYS DEC and get data from there to help find campsites. “I was pretty good at map and compass stuff from my years in Boy Scouts and always interested in environmental and land use issues”, he says.  Over time, he branched out into other geospatial areas including web services.  More recently He’s been focusing on more automation of processes, using Python and R statistical language to do some map plotting and a lot of Census data gathering and processing. “I like working with R as it is fast and easy to implement code in. I’ve also lately been doing a lot more with Leaflet and web services”.  Along the way he continues to use GeoPandas and Leaflet for map making. (btw as I was putting this blog piece together I found out the creator of Leaflet 11 yeas ago was  Volodymyr Agafonkin, a Ukrainian citizen who at the time was living in Kyiv.)  Content on the site is also made available in KMZ for use in Google Earth.

This is a example of how Arthur processed LIDAR data covering the Rome Sand Dunes west of the City of Rome in Oneida County. The landscape ifs a mosaic of sand dunes rising about 50 feet above low peat bogs which lie between the dunes. Processed LIDAR data renders the dunes very clearly. Arthur created this originally by writing a QGIS plugin that queries a shapefile with the LIDAR Digital Terrain Model Bare-Earthindex, then downloads the geotiffs, and finally joining them together to create the hillshade.The plugin itself is in Python and runs in QGIS, while the lidar download/processing script is in php-cli shell script.

The best place to start navigating the website is to open the table of contents link located in the upper right corner of the landing page. The table of contents page then provides additional links products and visuals Andy has created including aerial photos, charts, interactive maps (recommend starting here), and thematic maps to name just a few.   This page also provides more detail on open source components, some specifics on the use of Python and Pandas, a downloadable CSV file listing of web services (WMS, ArcGIS services, etc) used on the blog, and much more.  It’s worth noting that the website also includes non-GIS/geospatial content.

If you need some additional evidence of how much Arther has picked-up on programming, using open source components, and navigating the geospatial landscape in this space,  check out his tutorial on how to create a  Digital Surface Model GeoTIFF Using National Map Downloader, LiDAR Point Clouds and PDAL.  By example, the DSM image above is from a section of the Albany Pine Bush.  For a larger montage of the Albany Pine Bush digital surface model and samples of his code, click here for downloads.

And of course, the old stand-by hardcopy product. Here, a recently created thematic map of the City of Albany median year of housing construction map. He used the NYS Tax Parcel Centroid Points data aggregated down to the parcel level using R code and created a GeoPackage. Which was then used to create the map in QGIS. Additional layers were added for context.

There are many many more examples of geospatial products, maps, and viewers on the website.  Its a great example of how much can come out of the other end when diving into and applying geospatial tools to one’s own personal interests and way of living. 

When you have a few minutes over lunch or a cup of coffee, take a look at his site.  In communicating with Andy over the course of putting this piece together, he would be open to talking with and assisting non-profit or similar community groups on specific GIS/mapping projects.  His contact information is below. 

Contact:

Andy Arthur
www.andyarthur.org
andy@andyarthur.org

Geospatial Student Spotlight: Matthew Ward

Posted on by
Reply

Academic Institution:

Hunter College                                                          New York City
Department of Geography and Environmental Science
M.S. Geoinformatics (June 2021)

Virginia Tech                                                             Blacksburg, VA
School of Public and International Affairs
B.S. Environmental Policy and Planning   (May 2010)

Research Focus:

Ward’s graduate research at Hunter College focuses on the expanding use of Augmented Reality (AR) in the geospatial disciplines  with a  particular  interest  in  supporting civic  and public engagment.   AR is an interactive experience of the real-world environment where the objects that reside in the real world are enhanced by computer-generated content and perceptual information.   His research inspired by spatial AR research and development work at companies such as Google and  ESRI.

Ward’s work was recently highlighted and included  in the 2022 NYC OpenData  Week which included  augmenting street level imagery with the NYC PLUTO database which offers over 70 data variables at the tax parcel level.  Built on the WebXR standard, the web app leverages the THREE.js graphics library with files and front-end hosted on Github and the data on a Heroku  Postgres cloud platform.  The code repository and web app is available on Github. A short demo of the application can be seen here.

Cubes represent PLUTO centroids along the street face.   The magenta cube is the one currently selected by the user and provides current PLUTO information of the selected parcel in the upper left hand corner of the screen.  Cyan cubes represent other available PLUTO centroids for query. 

Ward’s smart phone app is now available (iOS and Android) here for entire NYC footprint.    Additional documentation and instructions are available on his GitHub repository.  

Selected Additional Geospatial Projects

Gerrymandering Web Map

Ward performed a series of analyses on the proposed plans by the New York State Independent Redistricting Commission (NYRIC) based on data related to the 2020 census.  Because the Commission couldn’t agree on a single draft plan, the bipartisan members drafted two plans.  Two of the analyses are shown below (References to the “Names” (Republican)”vs “Addresses” (Democrat) Plans are from the New York State Senate District plans submitted by the NYIRC.

The image above (from a web map) is a test of “roundness” (aka the Roeck test [1961]).   In general, rounder districts will have a higher Roeck score and oddly shaped or districts that are wider or longer (darker shaped districts) will receive lower scores. Although not a perfect assessment of gerrymandering, the Roeck test offers good comparison between district shapes. 

Another test done by Ward was comparing the average voter party tilt of a district to its neighboring districts (darker purple districts are more unlike their neighbors).  Metro NYC shows what could be considered large areas of voter homogeneity.

An map of an additional analysis is included with Ward’s paper (available here) summarizing this research work and findings.  The third review looked at identifying those proposed senate districts which had at least 30% minority group population (African-American, Asian, and Hispanic/Latino populations were used). Such districts are almost entirely located in New York City, with some inclusion of parts of Nassau county.  The research paper includes a comprehensive list of all data used in the gerrymandering analysis. A “live”  interactive version of the web mapping app can be accessed here.  The three different analysis for each plan (Names and Letters) can be viewed by clicking the layers button in the upper right hand corner of the viewing window.  (Designed for desktop viewers not smartphones.)

 Crime Modeling Across NYC

Ward’s graduate coursework also included developing a crime view modeling viewer (below) based on NYC OpenData reported crimes for 2019. Selected violent crime types were analyzed.  He developed an even square grid to cover the citywide footprint so as to render the spatio-temporal nature of the data.   Crimes that occur are tracked in space and time within those squares.  Data is associated with corresponding police precinct building addresses (how the data is made available through OpenData – not necessarily the exact crime location).

The data is for all of 2019 but each day of the model (365 days total) is rendering the three weeks of crime leading up to that day to predict ‘likelihood’ of crime going forward. Darker purple is an application “indicator” of where more violent crimes may occur based on the historical data.  Yellow grids meaning less likely than purple.  No color means no spatio-temporal connections in crimes or no crimes.  Click here to run the video.

Summary:

The Hunter College Masters in Geoinformatics (MSGEOi) focuses on the growing demand for emerging professionals  trained in the collection, organization, analysis, and dissemination of geospatial data.  Matthew Ward’s graduate work is illustrative of this as it intersects developing analytical methods and visualizations of large geospatial datasets. 

After school, Ward is interested in applying his work and research in the public or government space.  “I see the great power of combining open source technologies and open data portals for empowering local users with AR technology, he notes.  “For example, there are so many types of uses in helping the public visualize utilities and the public infrastructure or in environmental applications.”  

Augmented reality is no doubt a rapidly expanding and growing technology in the geospatial arena.

Contact:

Matthew Ward
Graduate Student
Hunter College, City University of New York (CUNY)
Department of Geography and Environmental Science
MATTHEW.WARD14@myhunter.cuny.edu

Dr. Sean Ahearn
Professor & Director, Center for Advanced Research of Spatial Information (CARSI)
Department of Geography and Environmental Science
sahearn@hunter.cuny.edu

Mapping a New Economy for Equality and Sustainability

Posted on by
Reply

New York City nonprofit uses GIS to help support cooperative and community-led development projects as well as advocacy campaigns

Introduction

Nonprofits are one of my favorite areas to cover in the geospatial space.  More than not their geospatial efforts operating on a shoestring budget, relying mostly on filtering through open data portals and mashing everything together to create visual products with open source software or via discounted vendor supported nonprofit software licensing agreements. Its not uncommon to find the person(s) doing spatial analysis and map making having been self-taught.  A lot gets done with a little.

One such organization is the New Economy Project (NEP) located in New York City.  NEP works with community groups to build a new economy that works for all, based on principles of cooperation, democracy, equity, racial justice, and ecological sustainability.  Their focused campaigns are carried out in concert with community, labor, civil rights and social justice groups on initiatives such as:

  • Public Bank NYC, a municipal public bank campaign rooted in economic and racial justice. Through public banking, NYC can divest from Wall Street banks, reinvest public deposits to support equitable and cooperative development.  
  • Advancing Community Land Trusts, through co-leadership of the NYC Community Land Initiative (NYCCLI) and one-on-one work with community groups seeking to organize community land trusts. 
  • Advancing NYS Community Equity Agenda, calling for transformation of the fundamentally unjust social, political and economic systems. The Equity Agenda puts racial and economic justice front and center and addresses long-standing inequities that play out at neighborhood and regional levels. 
  • Growing NYC’s Cooperative Economy through coalition-building and policy change that supports and brings together community land trusts; worker, housing, financial and food co-ops; and other models of cooperative, community-led development.

GIS and Mapping

Behind all of the mapping at the New Economy Project is Ben Hagen who currently serves as Senior Research and Communications Associate.  “As much of our work focuses financial services, a considerable amount of our data comes from the Federal Deposit Insurance Corporation (FDIC)” Ben notes, “and more specifically from the Summary of Deposits section.”   Ben does all of his mapping and data analysis with ArcMap having learned the craft as part of his graduate work in Urban Planning at New York University.  Other data sources frequently used by NEP include the U.S.Census Bureau, New York State Department of Financial Services,  New York State Office of Court Administration, and NYC OpenData. 

NEP uses use ESRI’s World Geocoding Service to support most of its address mapping though in the past had also used the city’s LION file. In addition to ArcGIS, the main other data-related software NEP uses is open source software R for cleaning data, especially when preparinfor for mapping. Hagen likes R software because it runs on text-based code and is great for replicating processes with minimal opportunities for introducing errors, as well as for easily sharing with others for a second look at methodology.  

Samples of NEP mapping products include (click image for map download).

(Left Image): Bank branches are not distributed and made accessible evenly across the city.  This map depict the relative absence of bank branches in  the Bronx, and the resulting concentration of high-cost alternatives such as check cashers and pawn shops. This is a series of maps that NEP updates every year covering most (if not all) of the city’s neighborhoods.  (Right Image): NEP creates maps showing the concentration of residential foreclosure risk in NYC communities of color. This uses data from the NYS Dept. of Financial Services, based on notices that mortgage servicers are required to send to delinquent borrowers 90 days before initiating a foreclosure action. While not all of these notices result in an eventual foreclosure, it provides a good indication of where the risk of home foreclosure is highest in the city.

(Left Image): Using data from the NYS Office of Court Administration, this NEP map highlights the concentration of debt collection lawsuits in communities of color and is included in this report prepared for AARP New York.  For additional maps, graphics and YouTube videos on this project visit this web page.  (Right Image):  Another map showing the relative absence of bank branches in NYC’s communities of color. Looking at this annually, one can see changes as gentrifying neighborhoods gain branches, but the overall pattern is a long-standing and remarkably stable one. There are actually six zip codes in the city, ranging in population from about 20,000 to 94,000 and all in communities of color, that have no bank branches at all.

Other examples of NEP research and mapping products can be viewed here.

Summary:

“The concept of “place” has always been at the core of how we approach our work” notes Hagen.  “In NYC, people feel very connected to their neighborhoods, and are acutely aware of how much the communities they call home shape their day-to-day lives. There’s no better way than maps to relate to people’s experience of their physical environment, and to help contextualize those environments within the larger, systemic frameworks that we are so often fighting to change.

Clearly, GIS is both an important business and commuication tool for NEP.

Contact:

Ben Hagen
Senior Research and Communications Associate
New Economy Project
121 W. 27th Street #804 | NY, NY 10001
212.680.5100, ext. 221 | neweconomynyc.org

 

 

 

 

 

NYC H2O: Advocating Cleaner Water through StoryMaps

Posted on by
Reply

NYC-based nonprofit has developed extensive portfolio of educational products for all ages and settings

StoryMaps are such a great communication tool.  Turning geographic and location-based features and concepts into an easy-to-understand medium for the much broader audience which help identify key community resources, assets and experiences that can inform planning and policy-making.  By combining narrative text and other multimedia content with Geographic Information System (GIS) maps and data, story maps serve as an effective means of helping residents and developers visualize and understand projects with greater clarity.  Increasingly becoming an art form unto itself.  Broadly and increasingly used across the geospatial spectrum from the industry-leading ESRI StoryMaps platform  to say the least of the versions being offered by others in the geospatial space including Google Earth, CARTO, or Tableau. 

I’m sure its not the first time Story Map subject matter has found its way to be included in a more technical oriented conference, but I was nonetheless interested, if not surprised, to see a presentation by NYC H2O at the recent the September 9th New York City Watershed Science and Technical Conference.   A conference historically the home for geospatial-based technical and scientific presentations by engineers, biologists, and others in the water science disciplines, NYC H2O’s presentation was entitled “Engaging an Urban Population With Water Engineering Utilizing a Virtual GIS Platform in the Pandemic Era”.

Having received its nonprofit 501(c)(3) designation in May, 2012, such work is not new to NYC H2O as it has built a niche in the environmental education space using the Story Map platform as its primary teaching and outreach tool. Story Maps augment and support their mission to “inspire and educate New Yorkers of all ages to learn about, enjoy and protect their city’s local water ecology.  Since 2009, NYC H2O has offered over 120 educational programs focused on NYC’s water system and ecology to a combined audience of 4,000 people.  Their StoryMaps often incorporate  digitized historic maps, photographs, videos, and social narratives to tell the story of water‐related  infrastructure.

A couple of NYC H20’s StoryMaps in the water sciences and environmental space include:

Water Systems Overview

I’ve often heard the New York City aqueduct system referred to as the “eight wonder of the world”.   Maybe not as sexy and visible as the Great Pyramid but an incredible engineering marvel nonetheless.  Conceived, designed and built in another era to support the drinking water lifeline of the city. 

NYC receives 90% of its water from the Delaware and Catskill Watersheds. Aqueducts (red line work) transports the fresh water from a series of reservoirs in the Catskill Mountains over 100-miles from the city.

Fitting that the first StoryMap posted by NYC H2O on their “Hub” website is entitled  Water Systems Overview outlining and showing the many geographies of the system which originates in the Catskills Region.  An interactive StoryMap highlighting the major aqueduct systems (Catskill and Delaware – named for the watersheds where the water is collected), as well as the individual reservoirs both upstate and those in the city.  Household water consumption stats are also included in the StoryMap as well as several interactive learning activities as illustrated in the application toolbar below.  Included in the Site Specific Lessons is a StoryMap just for High Bridge and the Jerome Park Reservoir.

Sewer System and Stormwater Management

This StoryMap, focusing on the relationship between storm water and sanitary sewage, and its overall impact on water quality in the metropolitan region has lots more maps and associated data.

Beginning in 1850, NYC laid 70 miles of sewers, or underground pipes that carried away wastewater.   Today the city has over 7,400 miles of sewer pipes. These pipes move wastewater with the help of 95 pumping stations to 14 sewage treatment plants located in all five boroughs.  Each day, NYC wastewater treatment plants process, or “clean”, a total of 1.3 billion gallons of wastewater.   Produceing about 1,400 tons, or 60 truckloads, of biosolids!

The city maintains 14 sewage treatment plants covering the five boroughs. Treated sewage effluent is discharged into the Hudson and East Rivers and Long Island Sound.

About 60% of New York City’s sewer system is a combined sewer system. This means that one single pipe carries both stormwater runoff AND sewage from city buildings.  During heavy rainstorms, combined sewers systems receive more stormwater runoff and sewage. Wastewater Treatment plants have a capacity, or a limit, to how much water they can clean. When there is more water than the capacity, a mix of stormwater and untreated sewage empties directly into the City’s surrounding waterways. These wet weather events are called combined sewer overflows (CSOs).

The presence of combined sewers are still very prevalent in the metropolitan region with the majority emptying into the East River and the east side of the Hudson River.

Also central to the long term management of clean water – are government regulated programs entitled Municipal Separate Storm Sewer System (MS4) programs.  In populated areas many neighborhoods, including in New York city, as well as industrial areas along coasts, have deployed MS4 programs.  A MS4 has one set of pipes that transports sewage to sewage treatment plants and another set of pipes that transports stormwater directly to local waterways.   Unfortunately, many pollutants (like oil, trash, pet waste, pesticides, and fertilizers from lawns) are washed into this system when it rains. MS4 helps solve one problem (CSOs and sewage in the local waterways) but causes a similar, although smaller, problem since trash and pollution can still enter the local waterways when it rains.

Currently, only about 40% of the NYC footprint is covered by an MS4.

As it relates to the collection of data for the purposes of monitoring citywide water quality, this StoryMap also includes interesting map displays of both fecal coliform (1985-2017) and dissolved oxygen (1909-2017) testing.

Summary:

It’s nice to see as part of such advocacy efforts the use of open data sets and geographic content as made available by government agencies and data portals.  Much of which is included in many of the 21 StoryMaps listed on the NYC H2O HUB.

“StoryMaps allowed us to keep students engaged in learning about their local water issues throughout the challenges of the pandemic. The tool has been so popular with schools that this year we launched a curriculum introducing GIS and StoryMaps to several environmental science and environmental justice in high school classrooms”, notes Kevin Barrett who guides NYC H2O’s mapping instruction.

With regard to the September 9th New York City Watershed Science and Technical Conference, Stalin Espinal, School Program Manager for NYC H2O adds “The conference gave us the opportunity to share our work and programs to a community of our peers. We were delighted to have received such enthusiasm and interest in our work with GIS and StoryMaps.  Even resulting in a new StoryMap collaboration in partnership with the Watershed Agriculture Council (WAC) which works with local farms on and around the upstate New York City watershed to promote Best-Practices and make improvements that help protect the water quality of our drinking water.”

Contact:

Kevin Barnett
GIS Education/Project Manager
kevin@nych2o.org
www.nych2o.org

LAMP: The Authoritative Liquor License Mapping Application for New York State

Posted on by
Reply

Its not too often one bumps into New York State Liquor Authority (SLA) GIS staff along the geospatial trail across the Empire State, but over there on the sidelines, agency staff continue to maintain an admirable web mapping application entitled  New York State Liquor Authority Mapping Project (LAMP).  I was able to catch up with William Cowley, Public Information Officer for the State Liquor Authority (SLA), to find out more about the 2021 release of LAMP.

Current LAMP Application

With the current version of the application finalized in March of this year, LAMP features an impressive tool kit to explore the vast statewide SLA database containing locational data on active, pending, and inactive permit data for beer, wine, and liquor establishments.  The data can be filtered even more specifically to show those locations which are categorized by:

  • On Premise Liquor
  • On Premise Wine
  • On Premise Beer
  • Off Premise Liquor
  • Off Premise Wine
  • Off Premise Beer
  • There are also permits for manufacturing and wholesale

In addition to serving as a great public information outreach tool, the application also serves as a useful utility and reference tool for applicants seeking a liquor license anywhere in the state.  The user online HELP documentation is excellent and available for download as well.

A 750-foot default search around 148 Martine Ave. in White Plains identifies 23 locations with active NYS liquor licenses, two pending, and 21 former or inactive licenses. Icons are colored coded to match active, pending, or inactive licenses keyed to the Legend box in the lower right-hand corner of the viewing screen (which can be expanded on the live online version).

Nearing almost ten years old, LAMP was originally launched in February 2012.  SLA staff assisted in the original development and launch of LAMP with consultant support from Fountains Spatial.  The 2021 LAMP update was completed with the assistance from Troy-based Cogent Technologies.  The viewing application is built on top of a customized version of ArcGIS Online (AGOL) and utilizes a SQL database to store SLA master data.  Updated daily, the application contains approximately 54,000 active and pending licenses.  The database is built on top of the application process which starts here.  The site makes use of the New York State Geocoding Service with the Google geocoding service as a backup.  LAMP is hosted by New York State Office for Information Technology Services.

Mapping Tools

The viewer includes an impressive user toolbox with several functions being AGOL defaults (i.e. scale dependency of map features, identify tools, .  Others have been customized to provide increased functionality and use of the data.  While changing base maps, scale dependency rendering of data, searching by address or user placed pin, or exporting search results to a CVS file are standard AGOL fare, LAMP offers some cool extended functions including, but not limited to:

    • Query results displayed as a new layer in map legend
    • Customized symbology for all feature types
    • A “measuring” tool which creates both a list of the closet (8) licensed facilities as well as within the 200’ and 500’ regulatory buffer areas (see below). A sample of a proximity report can be downloaded here.
    • An uber robust printing tool with advanced functions to create hardcopy maps
    • Easily include and attribute table view – below the main map window – of a particular layer

There are many other functions well documented and available in the application as well.

Image above highlights customized features in the LAMP application including expanded printing functions and creating new layers which are shown in the layer legend.

While the permitting process does not require the use of LAMP,  most applicants find it useful to identify prior licenses at their chosen location.  The locations of Public/Private Schools and Places of Worship are also included in the application because under the Alcoholic Beverage Control Law, SLA may not issue full liquor licenses (for bars/restaurants or for liquor stores) if the location is within 200-feet of a school or place of worship.  There is also a “500-foot law” which the agency must consider if there are already three or more active liquor licenses within 500 feet of the proposed location.

Integration with SLAM

In an interesting use and application of the SLA database, reference is made to BetaNYC’s SLAM viewer.  Used by community boards when reviewing liquor license applications and sidewalk cafe applications, the public facing tool aggregates information about active liquor licenses, sidewalk cafe licenses, 311 complaints about bars/restaurants/clubs, and restaurant health inspections in NYC onto a single map.   Statewide liquor licensing data can be downloaded from Open Data NY here.

SLAM effectively integrates numerous open data datasets including those from New York City and SLA. Users can zoom to particular addresses and click on relevant features to collect information that may be needed to review a license application. The tool is rendered through Carto – a Web mapping service

Contact:

William Crowley
Public Information Officer
New York State Liquor Authority
William.Crowley@sla.ny.gov

Hub2Hub: Sharing and Promoting the Geospatial Message

Posted on by
Reply

Publishing spatial data content has proven to be great outreach in terms of providing transparency and exposure for many government GIS programs – including here at Westchester County.  Increasingly much easier for geospatial organizations to do courtesy of powerful server technology allowing these same agencies to easily spin together data services based on common thematic features such as environmental, planimetric, business, demographics, cultural, parcels, and aerial photography to name only a few.  Though the data service environment is better understood and leveraged by a specific user community knowing what type of spatial content to look for, there remains even a larger community of geospatial users which navigate data portals to download just individual data sets.  Looking for that individual shapefile, KML, or .dwg file for their own individual project.

And even with all of this good geospatial content available, many publishing organizations – particularly government – struggle with building educational outreach programs for the larger GIS user community.  Tailored programs for users in community and advocacy groups, nonprofits, and even the business sector instructing how to effectively use and leverage our data.  Government GIS programs often do not have the resources or the business model to deliver these types of services outside of their organization.

Enter the emerging roll of “service” Hubs which specialize in leveraging government open data  in ways government programs normally cannot offer. Some are nonprofit in nature or affiliated with academic institutions.  Others, like OpenHub based in the lower Hudson Valley, is a for-profit venture.  The common theme among them is a window offering a wide range of engaging programs increasingly based on government data sets featuring hackathons, workshops and training programs,  application tools, and offering  business tech “round table-type” discussions.  While open source software such as OpenStreetMap, QGIS, Python, and R are commonly used to support their missions (and thus, the easy and affordable entry point for new users), one can see the use of ArcGIS Online and CARTO being offered as well.  Effectively expanding the use and understanding of geospatial concepts and bringing technology to the people.

Third Party “Hubs”

I’ve referenced and written before on the great work of BetaNYC serving in this space.  Particularly with regard to extending content of NYC OpenData  through their numerous programs including the flagship NYC School of Data conference.  While New York City does an incredible job in promoting and maintaining city government’s open data catalog itself, BetaNYC extends the effort by building an entire framework of putting the data into the hands of residents on the streets.

This BetaNYC viewing applications enables user to navigate the numerous political, administrative, and operational districts across New York City

Early in this space in the metro NYC area was the work of Steve Romalewski who started providing mapping services to the nonprofit sector as part of the Community Mapping Assistance Project (CMAP) which at the time was affiliated with New York Public Interest Research Group (NYPIRG).  Since 2006 he has been affiliated with the CUNY Mapping Service as part of the Center for Urban Research, Graduate Center at City University of New York (CUNY).

The Long Island Index was an early and very visible project that the CUNY Mapping helped create.  Another effort affiliated with the Center is the NYC Labor Market Information Service uses federal business and labor data sets.  Be sure to check out their Career Maps infographics and reports.  Similar academic efforts in the city exist at the Spatial Analysis and Visualization Initiative (SAVI) at Pratt Institute and the Center for Spatial Research at Columbia University.

Though originally developed at CMAP, the CUNY Mapping Service now hosts OASIS which is one of the most comprehensive mapping and viewing application covering the New York City footprint. Dozens of government generated datasets are made available in the viewer.

While the OpenHub is only beginning to jump into the geospatial space, its Founder Yulia Ovchinnikova will be highlighting the Westchester County GeoHub as part of the 2021 HV TechFest Conference as an example of open government and open-source data sharing to promote building communities. OpenHub will also be using content from the county’s GeoHub as part of a course project during a current online Data Analytics Bootcamp.  As regional technology and economic development is an important focus to OpenHub’s mission and purpose, our discussion started by focusing on a new web map being published which highlights the emerging BioSciences Ecosystem community in Westchester County.  The County is home to the largest biosciences cluster in New York State, boasting 8,000 jobs and 20 percent of the State’s total biosciences employment. The cluster comprises academic institutes doing basic research, R&D and clinical stage startups, large manufacturers and supply chain participants. OpenHub offers a mixture of both free and fee-based online programs including coding clubs, meetups, programming, and specific events for small businesses and startups. OpenHub is an open ecosystem resource center committed to building a tech-sector in the Hudson Valley and beyond

Government open data and economic development. BioScience business location X,Ys are available for download via ArcGIS Online.

MANY organizations leverage government generated geospatial data to support their own business needs and/or to champion a host of civic needs and issues.  In just one area, such as environmental justice (EJ), a Google search located a document published by the SeaGrant program at SUNY Stony Brook which identifies the numerous individual agencies (largely government and nonprofit) engaged in providing “Environmental Justice Mapping Tools for New York State Communities”.   Child care, social services, public safety, public transportation, sustainability, climate change, and civic issues only scratch the surface of the spaces where advocacy groups are involved in using government geospatial content for the larger public good.

Summary

What differentiates the groups itemized above are that they are more full service, a la carte organizations offering a wider range of products and services.  Staffing and resources to turn government data content into meaningful deliverables such as workshops (online and in-person), hackathon-type events, community events,  training courses, and even bringing the data to civic leaders and administrators in ways which is simply not possible by government publishing agencies.  Such outreach is often very targeted making the data more meaningful and ending up in the hands of those who understand its content the most.  Viewing and using the data at the “micro” as opposed to the “macro” or a larger geographic footprint.  Local is better.

While technically not all “hubs” in name, these groups continue to illustrate the growing benefit of how organizations are adding value and exposure to the government data assets.  In many scenarios, building partnerships with these types of service organizations can be of great value to government publishing agencies.

 

SPEED 2.0: Authoritative Environmental Remediation Mapping in New York City

Posted on by
Reply

Application Includes the use of both Open Source Software and Open Data Content

A lot of great geospatial projects and content are coming out of the NYC OpenData ecosystem.  In the same space  throughout the city is the deployment of applications and viewers using open source software.  One such app is the Searchable Property Environmental E-Database SPEED 2.0, built on top of CARTO and published by the Mayor’s Office of Environmental Remediation (OER).  I was introduced to the application via an online presentation organized by GISMO in March of this year.

SPEED 2.0 is an impressive collection of local/city, state, and federal geospatial datasets wrapped into one application for the purpose of helping individuals identify environmental issues – both current and past – on and/or adjacent to specific properties in New York City.  It is a sister application to the NYC Office of Environmental Remediation’s Environmental Project Information Center (EPIC) that provides information about the cleanup of brownfield sites across the city.

Individual parcels can be buffered by either 250’ or 500’ to show the proximity of adjacent parcels with current or past environmental issues, permitting, or contamination issues. Access to pertinent metadata is readily available.

Background

According to Lee Ilan, Chief of Planning in the Mayor’s Office of Environmental Remediation, the first version of SPEED was launched in 2009 as a web map with limited functionality and developed with PostGIS.  It was launched in support of the newly created office’s focus on the cleanup of brownfields across the city.  However, support for the initial application waned over the next several years with minimal new content added.  Post – SuperStorm Sandy provided new funding through the U.S. Housing and Urban Development (HUD) Community Development Block Grant Disaster Recovery Program which OER secured and offered the opportunity for a major rewrite and update of the original application. SPEED 2.0 was designed by their vendor Applied Geographics (AppGeo) to be a cloud-based application.  Originally the application was managed by the vendor but since December 2020, OER has assumed managing the app in the Google Cloud on their own.

The application also includes advanced search functions. For example, in the left-hand column using the filter options, I was able to identify only those OER projects in FEMA 100-year floodplains. Query is rendered in the map viewer.

“Carto software is helpful by providing a very modern user interface that generates layers which are compatible with Leaflet”, notes OER’s IT Director Maksim Kleban.  “It makes the transition from uploading our layers, and turning them into fully functional, interactive maps seamless.”  AppGeo proposed the use of CARTO to OER which has since found the software to be user friendly and simple to use with standalone online applications. Carto is licensed annually for the amount of space and resources needed for the SPEED application and works very similar to any other cloud solution, like Amazon Web Services, Microsoft Azure (AWS), or Google Cloud.

Currently there are about 50-55 datasets included in the SPEED viewer right now. The large majority are OER datasets which are updated automatically by syncing with data from external agencies’ datasets on Open Data, or from OER’s internal data sources.  Generally, they each have an independent update schedule which is also automated.   The data is managed mostly by automatic updates on OER’s server which communicates directly to Carto through an API. For layers which are not on an automatic update schedule, OER uses either a custom-designed interface or manually uploads data into Carto’s online platform.

User can search the SPEED database using a standardized address, common place names such as Bryant Park or Madison Square Garden (btw – even “MSG”!) or borough, block and lot (BBL) numbers.  The application also includes mark-up, feature transparency, and sharing tools,  great HELP documentation and easy access to metadata (as illustrated in the first image above) which is very helpful given the bevy of similar datasets from local, state and federal datasets accessible in the app.  Historical aerial photography from 1996, 1951, and 1924 enables users to identify previous land cover which can be an indicator of the presence of historic fill.  A “Sensitive Receptors“ layer includes the locations of facilities (schools, parks, libraries, health care, etc) where occupants are more susceptible to the effects of environmental contamination.

It continues to be a work in progress” says Ilan, “in the future we would like to also have functionalities for registered users. We also would like to add more analysis capabilities where new layers can be easily integrated with advanced search features”. 

SPEED 2.0 Featured on NYC Open Data Week

For the first time ever, OER participated in NYC Open Data Week in early March.  For those looking for a deeper dive into SPEED 2.0, use the link below to listen to Lee’s presentation.

Contact:

Ms. Lee Ilan
Chief of Planning
NYC Mayor’s Office of Environmental Remediation
lilan@cityhall.nyc.gov

Cruising Empire State Data Portals for Geospatial Content

Posted on by
Reply

Expanding Suite of Online Portals Offer Data For All Types of GIS Users

Ever since the federal government launched Data.gov in May 2009 and President Barack Obama issuing an executive order in May 2013 establishing the Open Data Policy, as well as   Governor Andrew M. Cuomo signing Executive Order 95 (“Using Technology to Promote Transparency, Improve Government Performance and Enhance Citizen Engagement”) in March of the same year, governments at all levels across the country have responded in creating an array of online data portals with the intent of making data free and easily accessible.  Available and open to everyone including nonprofits, application developers, researchers and academia, business and industry, service industries, the geospatial community and everything in-between. Such portals purpose and intent is to promote government transparency, reduce data request and distribution costs, fuel business development and job growth, and drive innovation.  Open data portals are particularly useful in times of crisis or major incident when multiple agencies require access to the same data and/or may be limited in context of  data sharing agreements either ad hoc or formal.

Current Empire State Data Portal Offerings

Today, geospatial data portals across the Empire State have a variety of aliases including Open, Gateway, Hub, even the old school Clearinghouse title.  For the purpose of this article we’ll take a look at a range of the current publically available sites focusing on current content and some of the general functionality as it relates to the GIS/geospatial user. Sites included are those currently available in New York City, state government, and at the county and municipal.  It is by no means inclusive of all the sites across the state though for the ones itemized below, take a drive and see what you find.

 Map Services vs. Files

Accessing, or consuming, geospatial content has changed significantly since the early years of data sharing.  The term “open data” was barely in the geospatial vernacular in old school GIS with many sites providing access to data which was governed by formal data sharing agreements and/or requiring some kind of organizational membership.  Offering file-based downloads to support, which at the time, was a heavy-client software-based world of GIS computing.

While file-based downloads from the myriad of portals across the Empire State are more than adequate for most users of the sites, GIS/geospatial users often have software enhanced capabilities to consume large volumes of data content in both client and browser applications via map services.   Consuming web services is now even an option within certain versions of AutoCAD which greatly expands the use of government authored and published geospatial content.

Given the option of having to take the time to download numerous individual files vs. consuming one map service with many data layers and picking only the data layers needed – the choice is often pretty obvious.  As such, data portals which offer geospatial content as a service (or by extension developer tools to provide similar functionality) increasingly have added value to the GIS/geospatial user.

New York State Government

HEALTH.DATA.NY.GOV & DATA.NY.GOV

Launched approximately around the same time in 2013,  these two portals provide lots of file based content for the GIS/geospatial community.  Built on top of Socrata software (Tyler Technologies) the two sites differ only in content. DATA.NY.GOV serving more as a generalized repository for state agencies (and selected local governments) and HEALTH.DATA.NY.GOV focusing specifically on public health related datasets.  Creating an user account (if needed) provides developers and publishers to identical tools and documentation to each site including customization options, database connections and API endpoints, metadata documentation, visualization tools, and more..  Both portals as part of the broader Open NY initiative.

While centralized data access and data visualization (charts, dashboards, etc) have always been a primary Socrata focus, the software also includes a basic toolkit  to map and render datasets which include coordinate geometry.  Both sites also include the ability to download datasets in a variety of formats including KML, KMZ, Shapefile, and GeoJSON which can be used by a wide range of GIS software platforms.

Users of DATA.NY.GOV site need to be aware this site ALSO includes content from other statewide Socrata sites (HEALTH.DATA.NY.GOV, NYC Open Data, and  Open Data Buffalo) so it is important to use the filtering function in the lower left corner of the page after the initial data theme/category has been selected.  For example, in the image below, after selecting the “Transportation” category, the application returns 521 records – which includes those meeting the same criteria from the other three statewide Socrata sites. (And in this category many from the NYC Socrata library.)   After filtering for “This site only” (DATA.NY.GOV) the application returns just 251 records.  A small nuance but filtering does help if the search is specific to certain jurisdictional or geographic footprints.  Also, when searching for state agency data across the available statewide portals, users will begin to see that some agency data is published and available on more than one portal.  For example, some state agency data can be found on  DATA.NY.GOV, CUGIR, and the state Clearinghouse.

The filtering function on DATA.NY.GOV limits the search to specific domains

Plenty of geospatial data in both of these state portals albeit as the GIS community has come to know during the COVID-19 pandemic, address or large scale data is hard come by in the public health space.  Respectfully a whole other discussion and until there are fundamental changes in how public health data is published, it will be difficult for non-public health agencies, advocacy groups, or the general public to access such data at units of geography smaller than the county level.  Or sometimes if one is lucky, finding some at the zip code level.

For additional information on how to export Socrata data to other formats and/or platforms see this help document.  Also, Koop software is an Open Geospatial ETL Engine which enables users to leave geospatial data where it lives and transform it into a variety of formats including GeoJSON, CSV, KML, a Shapefile, or Feature Service dynamically.

There is apparently ongoing work/research within the state GIS offices with regard to both mapping NY Open data using the Socrata JavaScript API as well as enabling the Socrata platform to use the NYS GIS Program Office geocoder.  If successful, such efforts will greatly expand the Socrata data portals for the geospatial community.

NY Department of State Geographic Information Gateway

Launched in 2015, the Gateway is administered out of the NY Department of State, Office of Planning, Development and Community Infrastructure and is now home to over 700 datasets. The Gateway is an online mapping application that makes it possible to view geographic information in its native viewer or Google Earth and users can also download information in multiple file formats or even pull into other online mapping applications.  While the original geographic footprint of the Gateway focused on New York’s offshore planning area in the ocean and State’s portion of the Great Lakes, the repository now contains a wide range of geospatial datasets covering the entire State.

The Gateway has datasets  grouped  thematically and made available as map services. All content includes accompanying metadata enabling users to identify which data layers to select and easily consume from within the composite map service.    Perfect for desktop or web client mash-ups and analysis.  The site also includes links, integration, and content with crowdsourced apps.

The Gateway is a must see site having itself apart from other NYS departments in context of building a geospatial data portal with so much functionality and content.  There is a good chance you’ll find content for your GIS project here.

The Gateway data browser has icons for actions that can be taken with desired datasets; download, view in native map viewer, export to Google Earth, information for ingesting as a map service and metadata.

 NYS GIS Clearinghouse

The Clearinghouse was originally implemented in the late 1990s to facilitate statewide data sharing.   For many years it was the defacto #1 portal for all things/files GIS.  The focus being based on formal/written data sharing agreements between government and non-profit organizations. Amazingly in 2021 these agreements are still part of the Clearinghouse administrative structure though it does not get in the way of non-data sharing members – even those from industry and business – from downloading many datasets.  Give credit to those individuals who helped create the original Clearinghouse as it was one of the first of its kind at the time.

Much has changed since the Clearinghouse was initially launched which has impacted the data component of the site.  Time itself, and in this case decades, has proven to be a major challenge.  Publishers have come and gone, numerous retirements and point of contacts no longer exist, and limited metadata and datasets being out-of-date.  It’s a next to impossible to manage and keep most of the data files up-to-date and uniquely different in portals like the Clearinghouse.  Given the new generation of geospatial data portals that  we should expect to continue to emerge (particularly at the local level), the relevancy gap of the Clearinghouse’s data file inventory will likely continue to widen.  To say the least of the role of other similar data and web services available online via other free government and business sites such as The National Map, EPA Envirofacts, and ESRI’s Living Atlas.

The upside is the Clearinghouse is emphasizing and pushing data content  as  web services having placed this “option” at the top of the data catalog list. Seemingly to encourage users to “try this first” before sitting down and mining through the catalog on a file-by-file basis.  Much of the web service content are from the go-to State GIS office program areas (imagery, elevation, addresses, geocoding services, and to a limited degree tax parcels) albeit other state agencies are contributing services as well.   Kudos to the state agencies are keeping their individual file catalogs up-to-date and current albeit as earlier mentioned some of these state agencies are placing data on multiple portals requiring users to search in more than one location.

Instead of performing file based searches, users can access content via map services from a variety of state agencies

Hopefully, the legal shackles of the Clearinghouse data sharing agreement – instituted in the previous century – alone do not continue to influence the long term focus of the Clearinghouse with regard to managing and publishing individual datasets. One day, the statewide geospatial community would probably be better served when all of the remaining relevant  Clearinghouse datasets are consolidated into DATA.NY.US or CUGIR.     And perhaps leave the Clearinghouse to focus just on publishing data as a service with the current capacity it has built.

Municipal Government

NYC Open Data

Another Socrata site, NYC Open Data is the big kid in the statewide open data sandbox with over 3,000 datasets grouped into just five categories available from nearly 100 reporting NYC departments.  In addition to just datasets, the site also includes files, documents, and external links.  So, if you are looking for geospatial data for the NYC geographic footprint this is the place to go.  The site has been around for several years already and is the creation of the  Mayor’s Office of Data Analytics (MODA) and the Department of Information Technology and Telecommunications (DoITT). As a hub of analytics in the City, MODA advocates for the use of Open Data in citywide data analytics and in the community.

The NYC Open Data Portal also provides excellent narratives and examples of how geospatial data is used and is critical to the operations of city government.

NYC Open Data exemplifies the use of open data giving rise to a broad user community including organizations such as BetaNYC which hosts events such as the NYC School of Data, Open Data Journeys, various Open Data Classes, and in concert with the Mayor’s Office, the signature NYC Open Data Week – this year March 6 – 14. And data from the portal is routinely used in the NYC BigApps competition.

To fully appreciate how far open data has come in NYC, take a look at the list of publications and reports on this web page which chronicles its growth over the past several years.  And if you need further convincing, take a look at the NYC Open Data Project Gallery.

Go to the site and drive around.   Its worth the ride.

City of Rochester (DataROC)

Cities are great units of geography to build data portals in that their governments often oversee management of all the geographic features and data within the municipal boundary footprint.  Albeit this does vary on occasion with regard to utilities and some infrastructure  Geospatial data is consistent at the city level as government business applications are built and deployed to capture data to the same geographic extent.  As city applications continue to be geo-enabled, and open data initiatives are initiated,  this geofencing of the data provides a consistent geographic extent and a great framework to support a geospatial data portal.

DataROC was launched in 2020 and due to the city’s long time relationship with ESRI and staff who are statisticians and/or business intelligence analysts.   “ESRI’s open data framework was a good fit for the city” according to Kate Day, Chief Performance Officer, Mayor’s Office of Data Analytics & Performance.   Prior to the rollout of DataROC, the city already had both a police data portal and a map gallery of different ESRI tools, so the first major part of DataROC was to bring previously decentralized content together and make it easy to search. GIS staff in the city’s IT department assisted as change champions to make sure that the wider GIS community spread out across the City were aware of what the focus of the portal was to be and drive more traffic to the  work analysts in various department had done over the years.  Currently there are 203 items in the city’s open data catalog containing data files, apps, maps, documents.  If you need geospatial data for a project inside the Rochester footprint, this is the portal to visit and search for data.

DataROC offers easy access to data, applications that feature maps curated for a specific purpose, featuring up-to-date data, and a gallery of web maps to get around the city

With regard to an “example” as how specific governments vet data to their portals, the City of Rochester “publishes anything that is classified as public data / documents / applications that have been requested to be published by an internal or external stakeholder, but only after they have been approved by a Data Governance Committee”.  This committee ensures each dataset is truly for public consumption (essentially that it poses no individual or group harm for being published, that it would not violate any law/regulation to publish, and that the dataset meets quality standards to accurately represent whatever subject is being depicted or summarized).

Other illustrative statewide local government data portals include the City of Buffalo (Open Data Buffalo), City of Albany (openAlbany),  Westchester County (GeoHub), and City of Syracuse (DataCuse).

Academic

Cornell University Geospatial Information Repository (CUGIR)

I’ve always been a huge fan of the CUGIR portal and related program offerings.  Housed in academia and on the fringe of the statewide GIS mainstream, its original homegrown portal was built  and developed in Java in the 1990s.   The current CUGIR portal was launched in January 2018 and  is is built on several modern open-source components.  The front-end interface uses GeoBlacklight — https://geoblacklight.org/  which is a collaborative project being developed by several universities, including NYU, Minnesota, Stanford, Princeton, UC Berkeley, Cornell, and others.  CUGIR has made various customizations, some of which have been incorporated back to the project.

Subjects such as landforms and topography, soils, hydrology, environmental hazards, agricultural activities, wildlife and natural resource management are included  in the CUGIR catalog. All data files are cataloged in accordance with FGDC standards and made available in widely used geospatial data formats.

Behind the scenes, CUGIR uses Solr for searching and faceting (also known as “data filters”).  The site  use GeoServer to access copies of the data in a PostGIS database to generate map previews via publically-accessible WMS/WFS web services for each dataset.  Amazon S3 is used to host all the downloadable data as pre-generated zipfiles, so downloads are quick and reliable. The site uses  Leaflet to display the map on top of a CARTO basemap (which is built from OpenStreetMap data the web viewer.   Maintaining and supporting the site is long-time CUGIR GIS front man, Keith Jenkins.

Today, CUGIR’s datasets total about 15GB.  The total number of “datasets” listed in the catalog is currently 458, which appears much smaller than the nearly 8000 it used to be because in the old system CUGIR had content split up into many smaller downloads to accommodate the bandwidth limitations of the late 1990s and early 2000s.  So to leverage newer technologies and internet speeds, several data series were merged into statewide datasets.  As part of the process, CUGIR also converted old formats like ArcInfo Interchange (.e00) files to shapefiles wherever possible.  As part of the site upgrade, old links which had bookmarked or published were updated as well.  Today, NYS DEC and USDA NRCS have the most CUGIR records with other large catalogs from NYS Department of Agriculture and Markets and Cornell Institute for Resource Information Sciences (IRIS).   Data is also available from nearby Tompkins County GIS.   Excellent filtering tools enable users to search data by  several different categories  and download datasets in Shapefile, PDF, or KML format and export to either KMZ or GeoJSON.

CUGIR focuses exclusively on geospatial data which sets it apart from other general sites like DATA.NY.GOV.  The NYS GIS Clearinghouse and the NYS Geographic Information Gateway are more similar to CUGIR with each having their own unique datasets which in some instances can be confusing for someone looking for data and not sure where to look.  Additional CUGIR functionality includes pointing to and maintaining an index map to external datasets such as the National Elevation Dataset (NED).    Jenkins and his CUGIR team hope to build out similar functionality and integration with other Empire State geospatial portals in the future.