Tag Archives: data visualization

Qntfii & Harkin Work Together to Provide Reality Capture Services and Education to the New York State Geospatial Community

Posted on by
Reply

The two New York State-based firms focus on drone and Lidar technology including SLAM (Simultaneous Localization and Mapping) applications.

The growth of UAV technology in New York State – particularly the Central New York (CNY) corridor is well documented and has the subject of eSpatiallyNewYork blog articles over the past couple years.  The region is home to a recognized UAV start-up incubator and there have been several news and market analysis articles covering the staggering growth of the industry, including a recent one (sample article available upon request) from Newswires. 

Two New York State firms in this space doing business together, albeit on the outskirts of the CYN region – one in Albany and the other Long Island:  Qntfill and Harkin.

History

The two initially crossed paths a couple years ago when tasked together on a Metropolitan Transportation Authority (MTA) project to map telecom infrastructure within the subway tunnels. However, the project was unfortunately scrapped before the two firms got to doing any actual (drone/laser/what kind) scanning – but it did serve as an introduction to each other’s services and laid the groundwork for many future collaborative efforts. 

Bill Gutelius is the principal at Qntfii who began his journey in the field of reality capture more than 25 years ago.  He transitioned from working with environmental engineering companies to working directly with the manufacturers of LIDAR equipment which at the time was a very emergent remote sensing technology.  He has founded a couple of startup business focused on LiDAR tools for reality capture, most recently starting Qntfi in 2014.  Qntfi is currently partnered with Emesent out of Brisbane Australia, the makers of the Hovermap SLAM LiDAR scanning tool.  Qntfii is an authorized Hovermap LiDAR sensing technology distributor.

Educated as an engineer, Scott Harrigan has served as President of Harkin Aerial since 2016 specializing in data visualization services to engineering, construction, and architectural clients, as well as custom engineered aerial solutions for a variety of industries.   His career path also included serving as a Product Specialist at Virtual Surveyor supporting clients on workflows to create quality CAD data from drone mapping.  He also served as Assistant Director of Long Island Group’s (LIG) Unmanned Aircraft Systems program, overseeing training, recruitment, logistics, and continuing education in UAS technologies for Civil Air Patrol members on Long Island.

Hovermap offers revolutionary advantages to challenging above ground, underground and GPS-denied environments. Underground mines, tunnels, bridges, telecom and transmission towers and many more can now be mapped and inspected safely, without putting staff at risk.

The different, but similar, individual focus and expertise uniquely augments their mutual business development.    Qntfii primarily focuses on handheld/ground level laser scanning, including indoors while Harkin brings drone and aerial data capture to the business equation.  Harrigan also coordinates training and education services as part of their joint work. Even though areas seem uniquely distinct, their services overlap on many projects.  One of their shared goals is to introduce drone, photogrammetry + LIDAR technology in a way that is both easy and cost effective for small-to-medium AEC (Architecture, Engineering, Construction) firms, as well as municipalities at the local town/village level to adopt. 

Selected Projects

Downtown Troy

This was a proof-of-concept project employing a truck-mounted Hovermap to collect street level 3D data of a portion of downtown Troy, NY at 4 AM.  This work was then followed by a couple of short drone flights during the day to collect well-lit imagery that was then processed in a standard photogrammetric software package.  The two datasets were then merged to colorize the LiDAR data from Hovermap. 

Cohoes Power Plant

Power plant operators were  interested in documenting the entire site (including a penstock tube) using LIDAR.   Hovermap played a critical role obtaining 3D data of parts of the plant which would have been impossible or very costly to collect.  In one application, Qntfill and Harking were able to place the Hovermap (secured in a protective aluminum cage) through an insertion point in the penstock tube and then lower it 50-60 feet inside the tube, mapping the geometry and condition of the penstock interior. 

Site Development

A DJI M300 drone with L1 LiDR was utilized by Harkin to produce bare-earth surface and contours for a new development in the Tri-State area.  Data was processed using DJI Terra (LiDAR) and Agisoft Metashape Photogrammetry software to provide up-to-date orthomosaics of the site.

By using the LIDAR-acquired surface, a 3D model was produced from excavation plans to determine the amount of cut and fill of earthwork required onsite.   The 3D model and volume calculations provided by Harkin allowed the site engineer to quickly adjust excavation plans to balance the earthwork onsite, greatly reducing the amount of trucks needed to bring in fill material, or take away excavated material. Cut/Fill calculations, contours, and surface were produced using Virtual Surveyor.  (Both images generated from Virtual Surveyor).

Harbors and Ports

In the Baltimore Harbor, U.S. Navy and commercial port operators were interested in knowing the condition of port/pier infrastructure as very little existing documentation (as-built plans) were available for facility planning purposes.  Qntfii flew a Hovermap drone to identify available space for vessels and equipment, as well as to establish the general configuration of the port(s) and arrangement of vessels currently berthed in the harbor.

Additionally, the two firms have combined on a wide-range of projects in government and industry including, but not limited to, the following: NYC Department of Environmental Protection (NYCDEP), New York City Economic Development Corporation (NYCEDC), Village of New Paltz, NY, Town of Oyster Bay, NY,  U.S. Marine Corps, U.S. Naval Information Warfare Systems Command (NAVWAR), and The New York Times.

“The drone market will most definitely continue to grow across the New York State, particularly as professional groups continue to come together and work together on common goals”, notes Harrigan who serves as Secretary for the NY chapter of AUVSI (Association for Unmanned Vehicle Systems International).  The chapter represents unmanned (drones, ROVs, robots) operators and businesses across the state and is probably the largest “drone” specific organization across the U.S.

Summary

“So much of this industry is geared towards “enterprise”,  big cities, and large – systems which requires a lot of manpower, IT, and intensive training to use. Our partnership focuses on bridging the gap to the small and mid-size organizations”, adds Harrigan.  

The two believe users don’t need to be large engineering/surveying firms or a state level department to have a successful drone and reality capture programs.  Both Gutelius and Harkin believe the best impact the UAV technology has is when it is working for smaller organizations.   

Contact

Bill Gutelius
Qntfill, Inc
1-518-406-0055
www.qntfii.co
info@qntfii.com

Scott Harrigan
Harkin Aerial
1-516-584-3035
www.harkin.io
info@harkin.io

Editor’s Note:  The 2024 IEEE Systems and Technologies for Remote Sensing Applications Through Unmanned Aerial Systems (STRATUS) is May 20-22, 2024 in Syracuse, New York

Mesonet Collects High-Accuracy Weather Data for Mapping and Data Visualization Applications

Posted on by
Reply

University at Albany Program Hosts Third Annual Symposium September 13-14

Background

In April 2014, the State of New York in collaboration with the Division of Homeland Security and Emergency Services funded the University at Albany to design, install, and operate the NYS Early Warning Weather Detection System. The centerpiece of the system is the New York State Mesonet (NYSM) network of 126 weather stations across the state, with at least one site in every county and borough. Each of the Mesonet’s 126 weather stations collects observations of surface temperature, relative humidity, wind speed and direction, precipitation, solar radiation, atmospheric pressure, snow depth, and soil moisture and temperature at three depths (5, 25, and 50 cm). Each site is also outfitted with a camera that collects still images.

In addition, several sub-networks of specialty sites have been deployed. A Profiler Network of 17 sites, which leverages LiDAR technology, provides additional atmospheric data in the vertical (up to 6 miles above ground); a Flux Network of 17 stations monitors the surface energy and a Snow Network of 20 sites which measures snow water content. Operated by the University at  Albany, the New York State Mesonet collects, archives, and processes data in real-time every five minutes, feeding weather prediction models and decision-support tools for users across the greater New York region. The NYS Mesonet implementation is part of the larger National Mesonet Program. NYSM is a cross-disciplinary team supporting a standard of excellence in delivering high-quality weather data across the Empire State.

The Mesonet network includes stations in all 62 New York State counties including the five boroughs of New York City

Metadata is provided for each site including lat/long, local landscape characteristics, pictures, elevation, soil classification, among other items.  Adjacent variables, such as obstructions, are reflected in a World Meteorological Organization (WMO) rating which indicates that the site’s surrounding environment may cause uncertainty in the data.

Detailed metadata is available for each station across the Mesonet network which can be used in supporting local weather-related mapping and analysis applications

Data and Geospatial Products

NYSM utilizes these high quality weather data to create end products designed for use by government, academic, and with commercial partners. NYSM supports solving real-world problems in agriculture, emergency management, energy, transportation and protecting public health. Much of the mapping and data visualization products on the Mesonet site is the responsibility of Nathan Bain, Sr. Software Engineer, NYS Mesonet, University at Albany, SUNY.  “The core set of software products I use in publishing content on the website include OpenLayers, python scripting, REACT for website development, and Highcharts for interactive time-series plots and line graphs.”

According to Bain, frequent users of the data are the National Weather Service, New York State Department of Health, New York State Department of Environment of Environmental Conservation, a variety of forensic meteorological companies, and private citizens.  Forty-four maps are updated every five minutes as presented on this page.

24-hour temp change is one of the many maps which is refreshed every five minutes on the Mesonet website

Mesonet field station data can be requested by completing and agreeing to the terms found in the Data Request Form. Data can be delivered in either NetCDF (network Common Data Form) which is a format for storing multidimensional scientific data (variables) such as temperature, humidity, pressure, wind speed, and direction. Each of the variables can be displayed through a dimension (such as time) in ArcGIS by making a layer or table view from the netCDF file. Data can also be delivered in normal CVS  format. Today’s weather statistics are available here. One interactive mapping application which Mesonet publishes “live” is its Transportation app showing the weather variables from sites along the NYS Thruway.

While many of the Mesonet Thruway weather stations are located in western New York near Buffalo, real-time data is collected as far east as Oneida.

September 2023 Conference

The Third Annual New York State (NYS) Mesonet Symposium will be held on 13-14 September 2023 at UAlbany’s ETEC building. Similar to previous symposiums, this will be a chance for people from academia, government, and the private sector to present on research and operational applications of the NYS Mesonet data. While in-person participation registration has been closed, those interested in participating via Zoom can use this link and a copy of the September agenda including presenters and participating organizations is available here.

Contact:

Nathan Bain
Software Engineer
NYS Mesonet
University at Albany
nbain@albany.edu

Developing and Applying the Geographic Aggregation Tool (GAT) at NYS Department of Health

Posted on by
Reply

“R”-based software application is used by public health researchers and data analysts

While often much of the Empire State GIS software discussion evolves around the use of the ESRI platform, there continues to be a steady dose and discussion of alternative geospatial client and web-based products. For example, staff at the New York State Department of Health (NYSDOH) use ArcGIS, but also other GIS programs including MapInfo, Map Marker, SatScan, and QGIS, as well as statistical programs like SAS, SPSS and Tableau for research and surveillance projects.

Supported by funding from the Centers for Disease Control and Prevention (CDC), staff from the NYSDOH Environmental Public Health Tracking (EPHT) Program – also known as “Tracking” – has leveraged the free and open-source software program “R” to develop and maintain the Geographic Aggregation Tool (GAT).  R has grown primarily out of the statistics and data analysis space which is very popular and used extensively in public health research.

GAT is currently maintained by Abigail Stamm at NYSDOH who supports a GAT webpage on GitHub containing an extensive listing of documentation, developer tools, metadata, tutorials and more.  According to Stamm, EPHT staff like using R for a couple of reasons including:

  • R is free,  easily accessible to the public, and enables staff to share the GAT package with agencies that do not have an ArcGIS license or the training and resources to use any of the other geographical software. (Legacy versions of GAT, including a SAS version, are archived on GitHub.)
  • The GAT package automates everything.  This is particularly valuable especially if users include scripting options which bypass the Graphic User Interface (series of pop-up windows) and reduces likelihood of mistakes and makes recording and reproducing workflows and updating much easier.

GAT documentation and content on GitHub.

At its core, GAT aggregates, or dissolves, geographic areas (most commonly used is census tract geography) based on numeric values for each area, such as case or population numbers, as well as other demographic values such as median income.  Health researchers often want subject data at a higher granularity than the county-level, which can conceal or mask variation, especially in counties with a mix of urban and rural populations.  Also, showing data at town level won’t work because many rural towns have very small populations.  Areas with small populations are likely to have few cases, resulting in unstable rates and also putting confidentiality of cases at risk.

To overcome these limitations NYSDOH developed GAT to join neighboring geographic areas together until a user defined population and/or number of cases is reached to support the statistical analysis desired. This allows local health departments and others to use rates to identify hot spots for targeted interventions. GAT can also be used to produce maps at varying geographic resolutions required by the user.

How GAT Works

GAT requests user inputs through a series of dialogs, including menus, checkboxes, and text boxes, so no programming knowledge is necessary.  GAT reads in a polygon shapefile which must contain, at minimum, a character variable that uniquely identifies areas and a numeric variable to sum for aggregation.  A series of dialog boxes allows the user to select:

  1. A variable to uniquely identify areas
  2. One or two aggregation variables
  3. Optionally, a variable of areas within which merging will be preferred (ex. county)
  4. The value (sum) to which the selected aggregation variable(s) should be aggregated
  5. The preferred aggregation method: closest geographic or population-weighted centroid, least value, or ratio of two values

Depending on the specifics of the data and the type of analysis of interest to the user, GAT offers four types of aggregation methods:

  1. Closest geographic centroid
  2. Closest population – weighted centroid
  3. Neighbor with the lowest count
  4. Most similar neighbor

Applying different rules or criteria to the different GAT aggregation results produce contrasting results. These are samples of the GAT aggregation tool when applied to total population numbers in towns in Hamilton and Fulton (NY) Counties. Sample code for producing these maps can be accessed here.

GAT produces two shapefiles, an aggregated file and a crosswalk. GAT also produces a PDF of maps and a log of the entire process, including user settings, any warnings, and a brief data dictionary. (The PDF and log provide much more information than being shown in this article).   These files are designed to help evaluate and report aggregation results and standardize user process.  NYSDOH staff developed GAT to standardize and automate how to aggregate New York’s 4900 census tracts.

Using the different aggregation methods may affect the values of resulting aggregated areas in different ways. For example, it was found when testing GAT that in cases where a small corner of a census tract contained most of its population, aggregating by geographic versus population weighted centroid could provide very different results. Also, when developing aggregated areas for the Tracking portal, it was recognized the portal would be displaying disease rates.  NYSDOH staff felt the most suitable aggregation method for these population-based measures would be to aggregate to the closest population weighted centroid.  To check for areas with smaller populations and unusually large numbers of cases, users have the option to aggregate by case count rather than population (or in addition to population).

While access to health data and resolution of data varies across the NYSDOH (point/address, census tract, zip code, or municipal level), staff in the Tracking program receive hospitalization and emergency room visit data at point level (resident address) from the Statewide Planning and Research Cooperative System (SPARCS). Staff geocode these datasets using multiple programs including MapMarker, SAM in ArcGIS, and NYCGBAT and assign the encompassing census tract to the point data. Methods to assign to tract based on zip code or town and to impute tracts for records that cannot be geocoded due to incomplete address have also been developed.

 Other Applications of GAT

Public Use in New York State

 Working towards NYSDOH’s goal of making sub-county data more accessible, NYS EPHT is developing a platform which will display environmental health outcomes and exposures in an interactive mapping application.  The current EPHT data portal displays county-level indicators but EPHT staff are redesigning it to include subcounty data using the sub-county aggregated areas created by GAT.  This will assist local agencies with targeting interventions while maintaining privacy and confidentiality.

Tracking is also working with Health Data NY (HDNY) to display the sub-county data. The HDNY platform will serve as a data repository displaying data for environmental health outcomes including asthma, chronic obstructive pulmonary disease (COPD), and myocardial infarction (MI; heart attacks), since these are the health outcomes local agencies are most interested in when planning extreme weather-related mitigation and resource allocation. The data will be available for public download along with the shapefile and a brief description of how the shapefile was developed.

The CDC is also using GAT to develop sub-county areas for various health outcome indicators having piloted it for data from other EPHT grantee states for health outcomes including asthma, MI, and COPD.

Sampling Design

 GAT is also being used in the development and selection of sampling areas as part of Biomonitoring NY, a statewide biomonitoring project.  In the first year of sampling this effort has focused on the sampling of households on Long Island. To this end, the group aggregated 2010 census block groups within census tracts so that each aggregated area had at least 440 households. In the next step, remaining tracts that were too small to meet the household minimum were aggregated to neighboring tracts. After completing the final aggregation, study staff randomly selected 25 aggregated areas and mailed postcards and invitation packets soliciting participants in households in the block groups within the selected aggregated areas.  Read more about the project.

Biomonitoring is a way of measuring the amount of environmental chemicals found in the human body. It is an important part of New York statewide epidemiological research that seeks to determine levels of chemical exposure in the human body and help better understand whether chemical exposures are associated with health effects in humans.

Informing Policy

In 2019, NYS passed the nation-leading Climate Leadership and Community Protection Act (CLCPA; aka “Climate Act”) to empower New York residents to fight climate change at home, at work, and in their communities.  NYSDOH, along with several other state agencies, is providing support and assisting in the implementation of the Climate Act. One effort includes identifying disadvantaged communities (DAC) that could benefit from mitigation efforts and allocation of investments. GAT was used to create sub-county areas for several climate-impacted health indicators that will be analyzed with multiple other variables to define DACs.

Learn More About GAT

A variety of documentation is available on the use and development of GAT.  Slides (pdf) from a 2020 and National Association of Health Data Organizations (NAHDO) presentation by NYSDOH staff are available here including a YouTube video.  GAT slides and a video from the 2021 useR Conference are also available.   Most recently staff presented at the 2021 Place and Health Conference in November which included epidemiologists, health geographers, social and behavioral scientists, statisticians, data scientists, and public health professionals from all levels of government.

GAT poster that was presented at the 2021 Place and Health Conference (download here). A lightning talk is also available.

Contact:

Abigail Stamm, Research Scientist
Bureau of Environmental and Occupational Epidemiology
New York State Department of Health
abigail.stamm@health.ny.gov

GAT was written in R-2.9.2 under Windows XP and was revised and converted to a package in R-3.4.3 under Windows 10 using RStudio-1.4.1103 and devtools-2.3.2. The latest version of GAT was compiled in R-3.6.1 and runs in R-3.5.3 through R-4.1.1.

https://github.com/ajstamm/gatpkg

Celebrating – and Visualizing – New York State Scenic Byways

Posted on by
Reply

Recent national designation highlights impressive Scenic Byways inventory across the Empire State

With little fanfare in January, the Federal Highway Administration, through the National Scenic Byways Program (NSBP), added the Palisades Parkway Scenic Byway to its list of New York State “national” designated Scenic Byways.  The Palisades designation – northern and southern most boundaries highlighted in the image below –  brings the Empire State’s National Scenic Byways total to four joining the Great Lakes Seaway Trail, Lakes to Locks Passage, and the Mohawk Towpath Byway.  These four combine for nearly 700 miles of nationally – designated scenic byways across the state.

The new nationally designated Palisades Parkway Scenic Byway, part of the State’s Scenic Byways program, originates in the south at the New York/New Jersey boundary heading north approximately 26.5 miles to just west of the Bear Mountain Bridge.

The designation augments the established New York State Scenic Byways program which includes both the federal and state designated scenic byways.  Administered through NYS Department of Transportation, the State’s Scenic Byway program was created in Highway Law (HAY 349-aa-dd) in 1995 authorizing the Commissioner of the NYSDOT to establish the scenic byways program and to coordinate the activities of state agencies, local governments and not-for-profit organizations.  Prior to the establishment of the state Scenic Byway program, the NYS Department of Environmental Conservation had a scenic roads program under Article 49 of the Environmental Conservation Law.  Those roads, along with all state parkways (listed in the regulations of the Commissioner of the Office of Parks, Recreation and Historic Preservation pursuant to Section 13.03 of Parks, Recreation and Historic Preservation Law) were incorporated into the Scenic Byway program under HAY 349-dd (k) and (l).

HAY 349-bb defines a “Scenic Byway” as a transportation route and adjacent area of particular scenic, historic, recreational, cultural or archeological characteristics which is managed to protect such characteristics and to encourage economic development through tourism and recreation.  Today, the State’s program includes the four national and 26 state designations.  A list of all designations the program is available here which includes a link to a thumbnail graphic of each byway.

State Scenic Byways Map

The January designation gave me reason to revisit a favorite piece of cartography I’ve been eyeing and wanting to write about for quite some time.  It’s a map on the Scenic Byways web page which speaks to the art of a well and purposely designed hardcopy map.  Increasingly lost as part of our insatiable appetite for interactive online mapping as the primary “cartographic” end product. Substituting the web viewer for what once was the primary GIS desktop objective:  The map.

Descriptive information and an individual map of each byway is found on the Scenic Byways homepage.

The map was created by staff in the Landscape Architecture Bureau (LAB) in the Department of Transportation. According to Christine Colley, Senior Landscape Architect and Scenic Byway Coordinator, its original creator was Marcy Sammons – who now works for another state agency.  Map revisions – to update routes and add new routes not previously included in the original map – is now the responsibility of Alexandra von Bieberstein.  Both staff are landscape architects whose background clearly contributed to the “feel” and overall simplistic cartographic design of the map. Symbology – colors, fonts, labels and shields – are all appropriately selected.   The state boundary clearly defined with green/gray green underlying base map colors which accentuates the colors representing the statewide byways. Even the limited use of selected hydrologic features – providing more context to the map than anything – as well as the limited amount of place names adds to the visual appeal of the map.  All map components combined to perfectly convey the statewide scenic byways story.

Colley notes “The map is created using ArcMap (LAB currently using version 10.7.1  although the original map may have been created using an earlier different version) and then then exported into Adobe Illustrator for editing and providing more detailed graphic enhancements.”

In absence of online mapping applications, Colley also adds that Scenic Byways staff respond to “map a specific Scenic Byway or to key locations on the Scenic Byway.  Staff also assists the public – using Google Maps to create a route – in navigating through the state to enable travelers to” requests by sending the public Google Maps “routes” from their location to:

  • Avoid parkways (which are limited access – no trucks, trailers or RVs on the parkways)
  • Encounter the most scenic route
  • Encounter the most scenic byways

For a myriad of reasons, hardcopy maps will continue to be an important product of the geospatial profession and a very special skillset to maintain.  The utility and importance of the traditional map has even been noted in online geospatial blogs and media publications in recent months.  Just last month there was an article in USA Today entitled ““Move aside, Google Maps, Apple Maps and GPS:  Why people still love their paper maps”

If you haven’t had a chance, download the map and take a look.  Pan around, zoom in to larger scales and take a spin around the state.

Enjoy the views.

Contact:

Christine Colley, RLA
Scenic Byway Coordinator
Senior Landscape Architect
New York State Department of Transportation
Office of the Environment
50 Wolf Rd, POD 4-1, Albany, NY  12232
518-485-9313 | christine.colley@dot.ny.gov
www.DOT.NY.gov

Green Map 2020: Evolving with Technology and Staying Community Focused

Posted on by
Reply

For nearly 30 years, Green Map has been an anchor in the Metro NYC nonprofit mapping community.  Led by its original founder, Wendy Brawer, Green Map’s portfolio of work and advocacy in locally-led mapping and sustainability efforts is quite impressive.  Since the launch of the Green Map System by Wendy Brawer in 1995 and publication of the original Green Map of NYC through her eco-design company, the program’s engagement tools and mapping resources have been adapted by locally-led projects in 65 countries.   Many joined the Green Map network when Greenhouse – their first content-managed website – was activated in 2007.   The site grew to become a rich resource highlighting community developed mapping work in all parts of the world.

While “green” mapping was one of its early defining products and focus,  the organization has expanded its services to include interactive web mapping, education, multimedia, event coordination, workshops, tours, planning and consulting, and creating software development tools.

 

Community Engagement Maps and Tools

Green Map’s award-winning icons, tools, platform(s) and tutorials help create engaging and visually powerful mapping content used as guides for local nature, culture, community living, social justice and resources for sustainable living. In addition to providing tools and tutorials on making maps, Green Map promotes their mapping efforts as part of Project Stories.  Project Stories are locally written and use tags to categorize the story type, theme, and purpose. Users can  discover how Green Map projects were created, people and organizations involved and about the project’s impact on their community.   Green Map projects are tagged and can be searched as part of the following 11 categories:

Besides its own mapping platform (with version 2 of the Open Green Map now in beta), the program also offers suggestions and tutorials on how to promote the use of the Green Map icons in other interactive mapping platforms such as ArcGIS Online (Story Maps), CARTO, Google Maps, Google Earth, and the Map of Tomorrow.

Continue reading →

COVID Era Geospatial Applications to Support Empire State Communities

Posted on by
Reply

Here in southeastern New York State, it’s amazing how fast the COVID wave came and went. At first, mid-March to mid-May seemed like an eternity, but now, not so much.  I likened the preparation for COVID to preparing for Super Storm Sandy.  Albeit phenomenally different, the buildup and the anxiety of waiting for it to show up.  Peering out the windows to see the outcome of a completely different kind of damage and suffering  altogether.  Physical and psychological.  Now July, it seems and feels like we can see the first wave of the pandemic in the rear view mirror.  But then as we refocus on the road in front of us, we know there are speed bumps ahead of us.  Not if, but when.

There were a plethora of interesting and meaningful geospatial viewers created across the state during the mid-March to mid-May time frame.  Most included highlighting COVID case numbers by larger units of geography (municipal or zip code boundaries),  generalized heat or pattern maps, impacts to businesses and institutions, as well as  information on hotlines and sources of assistance.  However, its pretty well documented the geospatial community across the state has struggled with being part of providing detailed spatial analysis or mapping support functions at the address level.   Or for that matter, even being asked to participate in this regard.

Personal health related data is uber protected with its use regulated and governed by the Health Insurance Portability and Accountability Act (HIPPA).  HIPPA  regulations reign supreme and contributed to a lesser involvement of GIS analysts in  providing geospatial content, metrics and analytics during the pandemic.  There is also the unspoken reluctance on the part of the public health community to release data sets  which might be difficult to understand or easily misinterpreted by the general public.  Department of Health  professionals often cautious to release data – particularly in new instances such as the COVID pandemic – and leaving analysis/interpretation  to internal professional staff only.  Ultimately though,  releasing some flavor of the data but in a more generalized context.

NYS Department of Health (DOH)  COVID-19 reporting has improved dramatically since the pandemic appeared in March.  Though while numbers  are made available only on a county basis, users can access and visualize the data in a variety of ways.  Reporting and posting of the data has become much more timely.  DOH is also now publishing a regional “Early Warning Monitoring Dashboard“.

Continue reading →

The Definitive Map for New York State Media Arts

Posted on by
Reply

New Online Application Shows Locations and Services for Organizations and Artists Across the State

Image from Heidi Neilson’s Southern Florida Beachball Antenna Archive (2017)

While the Empire State geospatial community is accustomed to seeing terms such as Google Maps JavaScript API, GeoJSON, and HTML as part of the conversation in building online mapping applications for government and business, we increasingly see the use of these same online mapping development tools in the nonprofit sector as well.  One such example is the newly updated New York Media Arts Map which is a project of of the New York State Council on the Arts (NYSCA) Electronic Media and Film program, managed by Wave Farm.  Though Wave Farm is more commonly recognized as an arts organization dedicated to transmission arts with FM and online radio stations, it is their regrant partnership through NYSCA which ultimately led to its stewardship of the online Media Arts Map. Serving in this capacity, Wave Farm assists in administering and distributing grant funds, ranging from $500 to $10,000, to dozens of New York State individual artists and organizations – all of which are included in the Map.

The Map

At its core, the Map is an extensive online catalogue of Media Artists, whose interdisciplinary work includes a wide range of fields including visual art, film, computer graphics, data visualization, sculpture, architecture, music, photography – even drones – and much more. The Map lists organizations which are recent grantees of the NYSCA Electronic Media and Film program (EMF) and the Media Arts Assistance Fund (MAAF).  In all, there are 198 organizations and 69 artists included in the Map.  First launched in 2009 and originally hosted by Rhizome, the 2014 update is the work of Peter Berry who is a developer affiliated with Supercosm, a Seattle, WA based firm which focuses on work with nonprofits.  In addition to the software components already mentioned, Peter’s work on the back end of the map includes the use of open source pieces CakePHP and Apache bundled with MySQL.

In addition to providing location information on organizations and artists included on the map, pull down menus along the top tool bar include additional statewide media arts information on meetings and workshops, partnership reports, and information on funding opportunities.

The Search box offers a wide range of options to filter the organization/artist database. Search queries provide a description of the types of work, services, and types of media arts offered as well as online contact information.

Intersection of Media Arts and Technology

It’s interesting to see, albeit even on some small level, how  the media arts world finds itself close to the geospatial space.  For example, take the work of  Harvestworks which supports the creation of art works achieved through the use of new and evolving technologies and provides an environment for experimentation with project consultants, technicians, instructors and innovative practitioners in all branches of the electronic arts.  A recent report by the organization entitled Common Ground provides “an overview of how and where transdisciplinary work is taking place in New York State, and makes recommendations to encourage further opportunities for artists that, when combined with environmental research, can aid engage the public, encourage environmental stewardship, and benefit communities”.

The report was published by the Harvestworks Digital Media Arts Center and The New York City Urban Field Station which is a partnership of the U.S. Forest Service, New York City Department of Parks and Recreation, and the Natural Areas Conservancy. (The Field Station in NYC supports the Arts and Humanities Residency Program. Check out the Summer 2018 Moon Arrow project – a time lapse compilation of the Moon Arrow pointing to the moon on New York City shorelines).  Open data, building code and hackathons in the geospatial space of your interest?  Then take a look the work of the School for Poetic Computation.  And how about drones?  While the technology is evolving incredibly fast, there was the 2018 Art Omi, Wave Farm and eteam live performance of The Aerial Actor which focused on the rising use of drones in science and art.  Actual drones were used as both carriers of messages and actors in the performance.  “The Map is a vital resource identifying art and technology projects and activity throughout New York State supported by NYSCA Electronic Media & Film”, says Galen Joseph-Hunter, Executive Director, Wave Farm. “It is also a useful tool for showing a geographic picture of how support is distributed throughout the state.”

Summary

So stop for a minute, grab a cup of coffee and put down the pen, I mean your mouse, and take a deep drive into the New York Media Arts Map site.  See what’s in your region, maybe even your neighborhood.  Support the statewide media arts community.  Cool stuff – all from the traditional artistic settings to the avant garde and experimental.  And sometimes, just a one-off from the geospatial space we work in.

Enjoy.

Contact:

Galen Joseph-Hunter
Executive Director
Wave Farm
Radio: WGXC 90.7-FM, Transmission Arts, Grants
518-622-2598
galen@wavefarm.org

Geospatial Business Spotlight: CARTO

Posted on by
Reply

Company Name:                   CARTO

Location:                               New York, New York​​​​​

Website:                               www.carto.com

Employees:                          143

Established:                         2009

Founded by Javier de la Torre, CARTO is a diverse and expanding company which includes data scientists, geospatial analysts, cartographers, software developers and engineers, visualization experts, and web designers focusing on Location Intelligence.  Most recently in May 2019, CARTO expanded its worldwide professional service portfolio offerings by acquiring Geographica.

Providing ready to use software tools for data scientists and application developers, CARTO’s client focus is on turning location data into business outcomes, and is built around the following workflow:

  • Data Ingestion & Management
  • Data Enrichment
  • Analysis
  • Solutions & Visualization
  • Integration

Software & Capabilities

Complex analysis, filtering, and visualization are integrated in real time reducing time-to-insight.  Users can integrate CARTO’s API’s and geocoding services to complement other apps and business applications and can be integrated with custom proprietary analytical models.  CARTO can be used as an engine to visualize a wide range of data services.

CARTO is scalable and offers a Software as a Service  (SaaS) deployment model to push new features instantly allowing users to “grow as you go.” Being enterprise-ready also means making on-premise and private clouds architecture solutions available to clients.  CARTO also offers a mobile platform.

Sample Products and Applications

On October 16, 2019, CARTO hosted the 2019 Spatial Data Science Conference (SDSC) at Columbia University which I covered and reported on in a previous blog post.  Typically GeoSpatial Business Spotlight focuses on three or four applications from the firm being highlighted.  However, since SDSC was a day-long series of entirely CARTO-based applications, the conference website provides a better and more thorough overview on how CARTO is applied in business, academia, government, and nonprofit organizations.  Choose from presentations by Uber, Facebook, University of Chicago, American Securities, Salesforce Maps, and MIT among others.  In Empire State, CARTO supports numerous programs in the metropolitan New York City area in both business and government.

Contributions to the Profession 

As part of CARTO’s long-standing commitment to FOSS, Open Source, Open Data, and Open Science, the company has collaborated with many organizations providing access to next generation geospatial technology, data, and models. Most recently (October 2019), CARTO’s Javier de la Torre  joined the Urban Computing Foundation (UCF) Technical Advisory Committee which is a neutral forum for accelerating geospatial open source and community development.  The UCF operates under the umbrella of The Linux Foundation.  In July 2019, Geospatial Media and Communications included Javier de la Torre as part of the Location Analytics & Business Intelligence (LA & BI) Advisory Board.  Additional

CARTO is an open source software built on PostGIS and PostgreSQL which was first released in Beta at FOSS4G in September 2011 and officially released at Where2.0 in April 2012.  The CARTO software solution uses JavaScript extensively in front end web applications, back end Node.js based APIs, and for client libraries.

Overall, CARTO’s platform consists of the following primary components:

The CARTO platform enables users to access and manage vast amounts of data while at the same time providing numerous processes to discover, filter, and integrate local and Big Data libraries.  Geo-enabling large datasets provides a means to visualize and better understand large and complex datasets. CARTO enriches user location data with versatile, relevant datasets, such as demographics and census, and advanced algorithms, drawn from CARTO’s own Data Observatory and offered as Data as a Service (DaaS).

CARTO uses widget-driven dashboards, an array of maps, and unified workflows so that non-GIS and non-mapping users/staff can bring the power of location into the organization’s decision making.

The CARTO software user interface provides both user-friendly mapping and dashboard visuals which can be customized to user needs and experience.

Complex analysis, filtering, and visualization are integrated in real time reducing time-to-insight.  Users can integrate CARTO’s API’s and geocoding services to complement other apps and business applications and can be integrated with custom proprietary analytical models.  CARTO can be used as an engine to visualize a wide range of data services.

CARTO is scalable and offers a Software as a Service  (SaaS) deployment model to push new features instantly allowing users to “grow as you go.” Being enterprise-ready also means making on-premise and private clouds architecture solutions available to clients.  CARTO also offers a mobile platform.

Contact:

Florence Broderick
VP Marketing
flo@carto.com
4475-686-89402

 

Spatial Data Science Conference 2019

Posted on by
Reply

I had the opportunity to attend the 2019 Spatial Data Science Conference (SDSC) at Columbia University on October 16th.  Hosted by NYC-based Carto, the event was attended by over 500 people from government, nonprofits, industry and business, and academia.  The day-long conference was highlighted by a variety of presentations and lightning talks from data scientists and program managers representing organizations from around the world including Uber, Airbnb, Datarobot, Waze, Instacart, MIT, Sidewalk Labs, Two Sigma, and Facebook among others.  All of the presenters use Carto’s Software as a Service (SaaS) platform which provides GIS functionality, web mapping, and spatial data science tools.

Data Science

Founded in 2017, SDSC  brings together organizations who are pushing the boundaries of spatial data modelling – ranging from large enterprise, to cities and government, as well as thought leaders from academic institutions.  Shown through the use of geospatial applications and organizational project initiatives, there was no doubt the common denominator and focus of those attending:  Data. It was definitely worth the trip.  SDSC is similar, but a very different kind of a “geospatial conference” for those of us who have spent a career running in traditional government geospatial circles.  Attendees and presenters are largely a completely different make-up from those normally attending the annual New York State GIS conferences.  (Of the 500+ preconference registrants, nearly 55% were from the private sector).   The day’s event included high quality presentations on the latest in modelling techniques, data science and analytics, visualization practices, and new data streams.  This later issue an increasingly important and interesting one across the statewide geospatial community as the day’s conversation clearly illustrated both the growing number of online geospatial data sources  (for example, numerous references were made to U.S. Census Bureau American Community Survey) and related data access tools.  Speakers noted both the importance and benefits of open data portals though not without the caveat that it was not uncommon to have to clean and often normalize the data prior to using in applications.  Numerous references to popular sites such as GitHub, Leaflet, OpenStreetMap, and Elastic (which presented at the event) were made with regard to supporting the open source ecosystem.

Mudit Srivastav from Australia-based Origin Energy, presented on the use of spatial data to support the increase sales of residential roof top solar panels. Note the many types of data the company is using to better define growth areas.

Data for the Social Good

Another common theme throughout the day was the use of geospatial data for the social good.  Interesting to hear the point being made not only from nonprofits and academia, but from the private sector as well.  Way far away from the normal Albany GIS crowd, Stuart Lynn made a presentation focusing on how Two Sigma, an investment management firm located in New York City, provides spatial analysis support through the company’s Data Clinic program to nonprofits, academic institutions, and government organizations.  Their focus:  Enabling and promoting social impact through data driven predictive models while funding breakthrough research, education and a wide range of charities and foundations.    The image below identifies some of their recent projects.  Great stuff and I’m already planning on a more in-depth article with Stuart in the future.

Enabling Social Impact Organizations with Spatial Analysis Techniques

Stuart’s talk, as well as others during the day, made reference to the  The Data for Good movement which was a social media movement first started by DataKind to highlight how data science could be used to help address a range of humanitarian issues. DataKind is a collaborative network of organizations that work together with data scientists to leverage the science of data for social impact.  DataKind’s afternoon presentation “Spatial Data Science for Social Good:  Improving Access to Dignified Sanitation in Haiti” was excellent.  The video for this session and all of the day’s presentations are now available online at the Spatial Data Science Conference website.

Arguably, the demand for “data scientists” will increasingly intersect and expand in government geospatial programs in areas such as the health and human services, climate change, public safety, sustainability and resiliency,  and social and environmental justice to name only a few.   As the universe of geospatial data continues to expand and be integrated with both new artificial intelligence (AI) and machine reading technologies, as well as combined with the availability of more powerful GIS software, it is easy to see where the demand for data scientists focusing on locational and predictive analytics is headed.

I asked Javier de la Torre, Carto founder and SDSC organizer, what his impressions were of the day’s presentations and content.  He replied:

“First, moving the data discussion from WHERE to Why. Time to move to analyzing data using maps opposed to just seeing data in maps.  Second, the rise of the Spatial Data Scientist and/or where advance GIS is emerging as a new platform. And third, identifying the need for better data marketplaces which provides interactive solutions resulting in increased performance to users”

Links to the 2017 and 2018 presentations are also available on the SDSC website.

Mobility for All Abilities Hackathon

Posted on by
Reply

While I personally did not attend the September 21st Mobility for All Abilities Hackathon at the  Brooklyn Army Terminal, I had originally intended to put together a short summary article to post on the day’s events and activities after gathering information from BetaNYC staff.  But instead, check out BetaNYC’s own detailed summary here – its quite good.

The event coincided with the 2019 National Day of Civic Hacking and was co-hosted by by BetaNYC and MakerSpace NYC which partnered with  NYC Department of Transportation, NYC Mayor’s Office of People with Disabilities, NYC Mayor’s Office of Data Analytics, and other organizations to setup eight challenges for a day-long map-a-thon and civic hacking to improve mobility equity in New York City.  Incredible results!

New York City Council Member Brad Lander provided opening remarks at the Mobility for All Abilities Hackathon. Photograph by Charlie Hartwell / CC BY

I’ve written about BetaNYC before and they have presented at the Westchester GIS User Group meeting in the past. They fill a much needed and recognized space in metro NYC towards facilitating these types of technology/geospatial events which can serve as a model for similar civic focused programs across the state.

For more information on their programs and activities, visit the BetaNYC website.