Mesonet Collects High-Accuracy Weather Data for Mapping and Data Visualization Applications

University at Albany Program Hosts Third Annual Symposium September 13-14

Background

In April 2014, the State of New York in collaboration with the Division of Homeland Security and Emergency Services funded the University at Albany to design, install, and operate the NYS Early Warning Weather Detection System. The centerpiece of the system is the New York State Mesonet (NYSM) network of 126 weather stations across the state, with at least one site in every county and borough. Each of the Mesonet’s 126 weather stations collects observations of surface temperature, relative humidity, wind speed and direction, precipitation, solar radiation, atmospheric pressure, snow depth, and soil moisture and temperature at three depths (5, 25, and 50 cm). Each site is also outfitted with a camera that collects still images.

In addition, several sub-networks of specialty sites have been deployed. A Profiler Network of 17 sites, which leverages LiDAR technology, provides additional atmospheric data in the vertical (up to 6 miles above ground); a Flux Network of 17 stations monitors the surface energy and a Snow Network of 20 sites which measures snow water content. Operated by the University at  Albany, the New York State Mesonet collects, archives, and processes data in real-time every five minutes, feeding weather prediction models and decision-support tools for users across the greater New York region. The NYS Mesonet implementation is part of the larger National Mesonet Program. NYSM is a cross-disciplinary team supporting a standard of excellence in delivering high-quality weather data across the Empire State.

The Mesonet network includes stations in all 62 New York State counties including the five boroughs of New York City

Metadata is provided for each site including lat/long, local landscape characteristics, pictures, elevation, soil classification, among other items.  Adjacent variables, such as obstructions, are reflected in a World Meteorological Organization (WMO) rating which indicates that the site’s surrounding environment may cause uncertainty in the data.

Detailed metadata is available for each station across the Mesonet network which can be used in supporting local weather-related mapping and analysis applications

Data and Geospatial Products

NYSM utilizes these high quality weather data to create end products designed for use by government, academic, and with commercial partners. NYSM supports solving real-world problems in agriculture, emergency management, energy, transportation and protecting public health. Much of the mapping and data visualization products on the Mesonet site is the responsibility of Nathan Bain, Sr. Software Engineer, NYS Mesonet, University at Albany, SUNY.  “The core set of software products I use in publishing content on the website include OpenLayers, python scripting, REACT for website development, and Highcharts for interactive time-series plots and line graphs.”

According to Bain, frequent users of the data are the National Weather Service, New York State Department of Health, New York State Department of Environment of Environmental Conservation, a variety of forensic meteorological companies, and private citizens.  Forty-four maps are updated every five minutes as presented on this page.

24-hour temp change is one of the many maps which is refreshed every five minutes on the Mesonet website

Mesonet field station data can be requested by completing and agreeing to the terms found in the Data Request Form. Data can be delivered in either NetCDF (network Common Data Form) which is a format for storing multidimensional scientific data (variables) such as temperature, humidity, pressure, wind speed, and direction. Each of the variables can be displayed through a dimension (such as time) in ArcGIS by making a layer or table view from the netCDF file. Data can also be delivered in normal CVS  format. Today’s weather statistics are available here. One interactive mapping application which Mesonet publishes “live” is its Transportation app showing the weather variables from sites along the NYS Thruway.

While many of the Mesonet Thruway weather stations are located in western New York near Buffalo, real-time data is collected as far east as Oneida.

September 2023 Conference

The Third Annual New York State (NYS) Mesonet Symposium will be held on 13-14 September 2023 at UAlbany’s ETEC building. Similar to previous symposiums, this will be a chance for people from academia, government, and the private sector to present on research and operational applications of the NYS Mesonet data. While in-person participation registration has been closed, those interested in participating via Zoom can use this link and a copy of the September agenda including presenters and participating organizations is available here.

Contact:

Nathan Bain
Software Engineer
NYS Mesonet
University at Albany
nbain@albany.edu

Developing and Applying the Geographic Aggregation Tool (GAT) at NYS Department of Health

“R”-based software application is used by public health researchers and data analysts

While often much of the Empire State GIS software discussion evolves around the use of the ESRI platform, there continues to be a steady dose and discussion of alternative geospatial client and web-based products. For example, staff at the New York State Department of Health (NYSDOH) use ArcGIS, but also other GIS programs including MapInfo, Map Marker, SatScan, and QGIS, as well as statistical programs like SAS, SPSS and Tableau for research and surveillance projects.

Supported by funding from the Centers for Disease Control and Prevention (CDC), staff from the NYSDOH Environmental Public Health Tracking (EPHT) Program – also known as “Tracking” – has leveraged the free and open-source software program “R” to develop and maintain the Geographic Aggregation Tool (GAT).  R has grown primarily out of the statistics and data analysis space which is very popular and used extensively in public health research.

GAT is currently maintained by Abigail Stamm at NYSDOH who supports a GAT webpage on GitHub containing an extensive listing of documentation, developer tools, metadata, tutorials and more.  According to Stamm, EPHT staff like using R for a couple of reasons including:

  • R is free,  easily accessible to the public, and enables staff to share the GAT package with agencies that do not have an ArcGIS license or the training and resources to use any of the other geographical software. (Legacy versions of GAT, including a SAS version, are archived on GitHub.)
  • The GAT package automates everything.  This is particularly valuable especially if users include scripting options which bypass the Graphic User Interface (series of pop-up windows) and reduces likelihood of mistakes and makes recording and reproducing workflows and updating much easier.

GAT documentation and content on GitHub.

At its core, GAT aggregates, or dissolves, geographic areas (most commonly used is census tract geography) based on numeric values for each area, such as case or population numbers, as well as other demographic values such as median income.  Health researchers often want subject data at a higher granularity than the county-level, which can conceal or mask variation, especially in counties with a mix of urban and rural populations.  Also, showing data at town level won’t work because many rural towns have very small populations.  Areas with small populations are likely to have few cases, resulting in unstable rates and also putting confidentiality of cases at risk.

To overcome these limitations NYSDOH developed GAT to join neighboring geographic areas together until a user defined population and/or number of cases is reached to support the statistical analysis desired. This allows local health departments and others to use rates to identify hot spots for targeted interventions. GAT can also be used to produce maps at varying geographic resolutions required by the user.

How GAT Works

GAT requests user inputs through a series of dialogs, including menus, checkboxes, and text boxes, so no programming knowledge is necessary.  GAT reads in a polygon shapefile which must contain, at minimum, a character variable that uniquely identifies areas and a numeric variable to sum for aggregation.  A series of dialog boxes allows the user to select:

  1. A variable to uniquely identify areas
  2. One or two aggregation variables
  3. Optionally, a variable of areas within which merging will be preferred (ex. county)
  4. The value (sum) to which the selected aggregation variable(s) should be aggregated
  5. The preferred aggregation method: closest geographic or population-weighted centroid, least value, or ratio of two values

Depending on the specifics of the data and the type of analysis of interest to the user, GAT offers four types of aggregation methods:

  1. Closest geographic centroid
  2. Closest population – weighted centroid
  3. Neighbor with the lowest count
  4. Most similar neighbor

Applying different rules or criteria to the different GAT aggregation results produce contrasting results. These are samples of the GAT aggregation tool when applied to total population numbers in towns in Hamilton and Fulton (NY) Counties. Sample code for producing these maps can be accessed here.

GAT produces two shapefiles, an aggregated file and a crosswalk. GAT also produces a PDF of maps and a log of the entire process, including user settings, any warnings, and a brief data dictionary. (The PDF and log provide much more information than being shown in this article).   These files are designed to help evaluate and report aggregation results and standardize user process.  NYSDOH staff developed GAT to standardize and automate how to aggregate New York’s 4900 census tracts.

Using the different aggregation methods may affect the values of resulting aggregated areas in different ways. For example, it was found when testing GAT that in cases where a small corner of a census tract contained most of its population, aggregating by geographic versus population weighted centroid could provide very different results. Also, when developing aggregated areas for the Tracking portal, it was recognized the portal would be displaying disease rates.  NYSDOH staff felt the most suitable aggregation method for these population-based measures would be to aggregate to the closest population weighted centroid.  To check for areas with smaller populations and unusually large numbers of cases, users have the option to aggregate by case count rather than population (or in addition to population).

While access to health data and resolution of data varies across the NYSDOH (point/address, census tract, zip code, or municipal level), staff in the Tracking program receive hospitalization and emergency room visit data at point level (resident address) from the Statewide Planning and Research Cooperative System (SPARCS). Staff geocode these datasets using multiple programs including MapMarker, SAM in ArcGIS, and NYCGBAT and assign the encompassing census tract to the point data. Methods to assign to tract based on zip code or town and to impute tracts for records that cannot be geocoded due to incomplete address have also been developed.

 Other Applications of GAT

Public Use in New York State

 Working towards NYSDOH’s goal of making sub-county data more accessible, NYS EPHT is developing a platform which will display environmental health outcomes and exposures in an interactive mapping application.  The current EPHT data portal displays county-level indicators but EPHT staff are redesigning it to include subcounty data using the sub-county aggregated areas created by GAT.  This will assist local agencies with targeting interventions while maintaining privacy and confidentiality.

Tracking is also working with Health Data NY (HDNY) to display the sub-county data. The HDNY platform will serve as a data repository displaying data for environmental health outcomes including asthma, chronic obstructive pulmonary disease (COPD), and myocardial infarction (MI; heart attacks), since these are the health outcomes local agencies are most interested in when planning extreme weather-related mitigation and resource allocation. The data will be available for public download along with the shapefile and a brief description of how the shapefile was developed.

The CDC is also using GAT to develop sub-county areas for various health outcome indicators having piloted it for data from other EPHT grantee states for health outcomes including asthma, MI, and COPD.

Sampling Design

 GAT is also being used in the development and selection of sampling areas as part of Biomonitoring NY, a statewide biomonitoring project.  In the first year of sampling this effort has focused on the sampling of households on Long Island. To this end, the group aggregated 2010 census block groups within census tracts so that each aggregated area had at least 440 households. In the next step, remaining tracts that were too small to meet the household minimum were aggregated to neighboring tracts. After completing the final aggregation, study staff randomly selected 25 aggregated areas and mailed postcards and invitation packets soliciting participants in households in the block groups within the selected aggregated areas.  Read more about the project.

Biomonitoring is a way of measuring the amount of environmental chemicals found in the human body. It is an important part of New York statewide epidemiological research that seeks to determine levels of chemical exposure in the human body and help better understand whether chemical exposures are associated with health effects in humans.

Informing Policy

In 2019, NYS passed the nation-leading Climate Leadership and Community Protection Act (CLCPA; aka “Climate Act”) to empower New York residents to fight climate change at home, at work, and in their communities.  NYSDOH, along with several other state agencies, is providing support and assisting in the implementation of the Climate Act. One effort includes identifying disadvantaged communities (DAC) that could benefit from mitigation efforts and allocation of investments. GAT was used to create sub-county areas for several climate-impacted health indicators that will be analyzed with multiple other variables to define DACs.

Learn More About GAT

A variety of documentation is available on the use and development of GAT.  Slides (pdf) from a 2020 and National Association of Health Data Organizations (NAHDO) presentation by NYSDOH staff are available here including a YouTube video.  GAT slides and a video from the 2021 useR Conference are also available.   Most recently staff presented at the 2021 Place and Health Conference in November which included epidemiologists, health geographers, social and behavioral scientists, statisticians, data scientists, and public health professionals from all levels of government.

GAT poster that was presented at the 2021 Place and Health Conference (download here). A lightning talk is also available.

Contact:

Abigail Stamm, Research Scientist
Bureau of Environmental and Occupational Epidemiology
New York State Department of Health
abigail.stamm@health.ny.gov

GAT was written in R-2.9.2 under Windows XP and was revised and converted to a package in R-3.4.3 under Windows 10 using RStudio-1.4.1103 and devtools-2.3.2. The latest version of GAT was compiled in R-3.6.1 and runs in R-3.5.3 through R-4.1.1.

https://github.com/ajstamm/gatpkg

Celebrating – and Visualizing – New York State Scenic Byways

Recent national designation highlights impressive Scenic Byways inventory across the Empire State

With little fanfare in January, the Federal Highway Administration, through the National Scenic Byways Program (NSBP), added the Palisades Parkway Scenic Byway to its list of New York State “national” designated Scenic Byways.  The Palisades designation – northern and southern most boundaries highlighted in the image below –  brings the Empire State’s National Scenic Byways total to four joining the Great Lakes Seaway Trail, Lakes to Locks Passage, and the Mohawk Towpath Byway.  These four combine for nearly 700 miles of nationally – designated scenic byways across the state.

The new nationally designated Palisades Parkway Scenic Byway, part of the State’s Scenic Byways program, originates in the south at the New York/New Jersey boundary heading north approximately 26.5 miles to just west of the Bear Mountain Bridge.

The designation augments the established New York State Scenic Byways program which includes both the federal and state designated scenic byways.  Administered through NYS Department of Transportation, the State’s Scenic Byway program was created in Highway Law (HAY 349-aa-dd) in 1995 authorizing the Commissioner of the NYSDOT to establish the scenic byways program and to coordinate the activities of state agencies, local governments and not-for-profit organizations.  Prior to the establishment of the state Scenic Byway program, the NYS Department of Environmental Conservation had a scenic roads program under Article 49 of the Environmental Conservation Law.  Those roads, along with all state parkways (listed in the regulations of the Commissioner of the Office of Parks, Recreation and Historic Preservation pursuant to Section 13.03 of Parks, Recreation and Historic Preservation Law) were incorporated into the Scenic Byway program under HAY 349-dd (k) and (l).

HAY 349-bb defines a “Scenic Byway” as a transportation route and adjacent area of particular scenic, historic, recreational, cultural or archeological characteristics which is managed to protect such characteristics and to encourage economic development through tourism and recreation.  Today, the State’s program includes the four national and 26 state designations.  A list of all designations the program is available here which includes a link to a thumbnail graphic of each byway.

State Scenic Byways Map

The January designation gave me reason to revisit a favorite piece of cartography I’ve been eyeing and wanting to write about for quite some time.  It’s a map on the Scenic Byways web page which speaks to the art of a well and purposely designed hardcopy map.  Increasingly lost as part of our insatiable appetite for interactive online mapping as the primary “cartographic” end product. Substituting the web viewer for what once was the primary GIS desktop objective:  The map.

Descriptive information and an individual map of each byway is found on the Scenic Byways homepage.

The map was created by staff in the Landscape Architecture Bureau (LAB) in the Department of Transportation. According to Christine Colley, Senior Landscape Architect and Scenic Byway Coordinator, its original creator was Marcy Sammons – who now works for another state agency.  Map revisions – to update routes and add new routes not previously included in the original map – is now the responsibility of Alexandra von Bieberstein.  Both staff are landscape architects whose background clearly contributed to the “feel” and overall simplistic cartographic design of the map. Symbology – colors, fonts, labels and shields – are all appropriately selected.   The state boundary clearly defined with green/gray green underlying base map colors which accentuates the colors representing the statewide byways. Even the limited use of selected hydrologic features – providing more context to the map than anything – as well as the limited amount of place names adds to the visual appeal of the map.  All map components combined to perfectly convey the statewide scenic byways story.

Colley notes “The map is created using ArcMap (LAB currently using version 10.7.1  although the original map may have been created using an earlier different version) and then then exported into Adobe Illustrator for editing and providing more detailed graphic enhancements.”

In absence of online mapping applications, Colley also adds that Scenic Byways staff respond to “map a specific Scenic Byway or to key locations on the Scenic Byway.  Staff also assists the public – using Google Maps to create a route – in navigating through the state to enable travelers to” requests by sending the public Google Maps “routes” from their location to:

  • Avoid parkways (which are limited access – no trucks, trailers or RVs on the parkways)
  • Encounter the most scenic route
  • Encounter the most scenic byways

For a myriad of reasons, hardcopy maps will continue to be an important product of the geospatial profession and a very special skillset to maintain.  The utility and importance of the traditional map has even been noted in online geospatial blogs and media publications in recent months.  Just last month there was an article in USA Today entitled ““Move aside, Google Maps, Apple Maps and GPS:  Why people still love their paper maps

If you haven’t had a chance, download the map and take a look.  Pan around, zoom in to larger scales and take a spin around the state.

Enjoy the views.

Contact:

Christine Colley, RLA
Scenic Byway Coordinator
Senior Landscape Architect
New York State Department of Transportation
Office of the Environment
50 Wolf Rd, POD 4-1, Albany, NY  12232
518-485-9313 | christine.colley@dot.ny.gov
www.DOT.NY.gov

Green Map 2020: Evolving with Technology and Staying Community Focused

For nearly 30 years, Green Map has been an anchor in the Metro NYC nonprofit mapping community.  Led by its original founder, Wendy Brawer, Green Map’s portfolio of work and advocacy in locally-led mapping and sustainability efforts is quite impressive.  Since the launch of the Green Map System by Wendy Brawer in 1995 and publication of the original Green Map of NYC through her eco-design company, the program’s engagement tools and mapping resources have been adapted by locally-led projects in 65 countries.   Many joined the Green Map network when Greenhouse – their first content-managed website – was activated in 2007.   The site grew to become a rich resource highlighting community developed mapping work in all parts of the world.

While “green” mapping was one of its early defining products and focus,  the organization has expanded its services to include interactive web mapping, education, multimedia, event coordination, workshops, tours, planning and consulting, and creating software development tools.

 

Community Engagement Maps and Tools

Green Map’s award-winning icons, tools, platform(s) and tutorials help create engaging and visually powerful mapping content used as guides for local nature, culture, community living, social justice and resources for sustainable living. In addition to providing tools and tutorials on making maps, Green Map promotes their mapping efforts as part of Project Stories.  Project Stories are locally written and use tags to categorize the story type, theme, and purpose. Users can  discover how Green Map projects were created, people and organizations involved and about the project’s impact on their community.   Green Map projects are tagged and can be searched as part of the following 11 categories:

Besides its own mapping platform (with version 2 of the Open Green Map now in beta), the program also offers suggestions and tutorials on how to promote the use of the Green Map icons in other interactive mapping platforms such as ArcGIS Online (Story Maps), CARTO, Google Maps, Google Earth, and the Map of Tomorrow.

Continue reading

COVID Era Geospatial Applications to Support Empire State Communities

Here in southeastern New York State, it’s amazing how fast the COVID wave came and went. At first, mid-March to mid-May seemed like an eternity, but now, not so much.  I likened the preparation for COVID to preparing for Super Storm Sandy.  Albeit phenomenally different, the buildup and the anxiety of waiting for it to show up.  Peering out the windows to see the outcome of a completely different kind of damage and suffering  altogether.  Physical and psychological.  Now July, it seems and feels like we can see the first wave of the pandemic in the rear view mirror.  But then as we refocus on the road in front of us, we know there are speed bumps ahead of us.  Not if, but when.

There were a plethora of interesting and meaningful geospatial viewers created across the state during the mid-March to mid-May time frame.  Most included highlighting COVID case numbers by larger units of geography (municipal or zip code boundaries),  generalized heat or pattern maps, impacts to businesses and institutions, as well as  information on hotlines and sources of assistance.  However, its pretty well documented the geospatial community across the state has struggled with being part of providing detailed spatial analysis or mapping support functions at the address level.   Or for that matter, even being asked to participate in this regard.

Personal health related data is uber protected with its use regulated and governed by the Health Insurance Portability and Accountability Act (HIPPA).  HIPPA  regulations reign supreme and contributed to a lesser involvement of GIS analysts in  providing geospatial content, metrics and analytics during the pandemic.  There is also the unspoken reluctance on the part of the public health community to release data sets  which might be difficult to understand or easily misinterpreted by the general public.  Department of Health  professionals often cautious to release data – particularly in new instances such as the COVID pandemic – and leaving analysis/interpretation  to internal professional staff only.  Ultimately though,  releasing some flavor of the data but in a more generalized context.

NYS Department of Health (DOH)  COVID-19 reporting has improved dramatically since the pandemic appeared in March.  Though while numbers  are made available only on a county basis, users can access and visualize the data in a variety of ways.  Reporting and posting of the data has become much more timely.  DOH is also now publishing a regional “Early Warning Monitoring Dashboard“.

Continue reading

The Definitive Map for New York State Media Arts

New Online Application Shows Locations and Services for Organizations and Artists Across the State

Image from Heidi Neilson’s Southern Florida Beachball Antenna Archive (2017)

While the Empire State geospatial community is accustomed to seeing terms such as Google Maps JavaScript API, GeoJSON, and HTML as part of the conversation in building online mapping applications for government and business, we increasingly see the use of these same online mapping development tools in the nonprofit sector as well.  One such example is the newly updated New York Media Arts Map which is a project of of the New York State Council on the Arts (NYSCA) Electronic Media and Film program, managed by Wave Farm.  Though Wave Farm is more commonly recognized as an arts organization dedicated to transmission arts with FM and online radio stations, it is their regrant partnership through NYSCA which ultimately led to its stewardship of the online Media Arts Map. Serving in this capacity, Wave Farm assists in administering and distributing grant funds, ranging from $500 to $10,000, to dozens of New York State individual artists and organizations – all of which are included in the Map.

The Map

At its core, the Map is an extensive online catalogue of Media Artists, whose interdisciplinary work includes a wide range of fields including visual art, film, computer graphics, data visualization, sculpture, architecture, music, photography – even drones – and much more. The Map lists organizations which are recent grantees of the NYSCA Electronic Media and Film program (EMF) and the Media Arts Assistance Fund (MAAF).  In all, there are 198 organizations and 69 artists included in the Map.  First launched in 2009 and originally hosted by Rhizome, the 2014 update is the work of Peter Berry who is a developer affiliated with Supercosm, a Seattle, WA based firm which focuses on work with nonprofits.  In addition to the software components already mentioned, Peter’s work on the back end of the map includes the use of open source pieces CakePHP and Apache bundled with MySQL.

In addition to providing location information on organizations and artists included on the map, pull down menus along the top tool bar include additional statewide media arts information on meetings and workshops, partnership reports, and information on funding opportunities.

The Search box offers a wide range of options to filter the organization/artist database. Search queries provide a description of the types of work, services, and types of media arts offered as well as online contact information.

Intersection of Media Arts and Technology

It’s interesting to see, albeit even on some small level, how  the media arts world finds itself close to the geospatial space.  For example, take the work of  Harvestworks which supports the creation of art works achieved through the use of new and evolving technologies and provides an environment for experimentation with project consultants, technicians, instructors and innovative practitioners in all branches of the electronic arts.  A recent report by the organization entitled Common Ground provides “an overview of how and where transdisciplinary work is taking place in New York State, and makes recommendations to encourage further opportunities for artists that, when combined with environmental research, can aid engage the public, encourage environmental stewardship, and benefit communities”.

The report was published by the Harvestworks Digital Media Arts Center and The New York City Urban Field Station which is a partnership of the U.S. Forest Service, New York City Department of Parks and Recreation, and the Natural Areas Conservancy. (The Field Station in NYC supports the Arts and Humanities Residency Program. Check out the Summer 2018 Moon Arrow project – a time lapse compilation of the Moon Arrow pointing to the moon on New York City shorelines).  Open data, building code and hackathons in the geospatial space of your interest?  Then take a look the work of the School for Poetic Computation.  And how about drones?  While the technology is evolving incredibly fast, there was the 2018 Art Omi, Wave Farm and eteam live performance of The Aerial Actor which focused on the rising use of drones in science and art.  Actual drones were used as both carriers of messages and actors in the performance.  “The Map is a vital resource identifying art and technology projects and activity throughout New York State supported by NYSCA Electronic Media & Film”, says Galen Joseph-Hunter, Executive Director, Wave Farm. “It is also a useful tool for showing a geographic picture of how support is distributed throughout the state.”

Summary

So stop for a minute, grab a cup of coffee and put down the pen, I mean your mouse, and take a deep drive into the New York Media Arts Map site.  See what’s in your region, maybe even your neighborhood.  Support the statewide media arts community.  Cool stuff – all from the traditional artistic settings to the avant garde and experimental.  And sometimes, just a one-off from the geospatial space we work in.

Enjoy.

Contact:

Galen Joseph-Hunter
Executive Director
Wave Farm
Radio: WGXC 90.7-FM, Transmission Arts, Grants
518-622-2598
galen@wavefarm.org

Geospatial Business Spotlight: CARTO

Company Name:                   CARTO

Location:                               New York, New York​​​​​

Website:                               www.carto.com

Employees:                          143

Established:                         2009

Founded by Javier de la Torre, CARTO is a diverse and expanding company which includes data scientists, geospatial analysts, cartographers, software developers and engineers, visualization experts, and web designers focusing on Location Intelligence.  Most recently in May 2019, CARTO expanded its worldwide professional service portfolio offerings by acquiring Geographica.

Providing ready to use software tools for data scientists and application developers, CARTO’s client focus is on turning location data into business outcomes, and is built around the following workflow:

  • Data Ingestion & Management
  • Data Enrichment
  • Analysis
  • Solutions & Visualization
  • Integration

Software & Capabilities

Complex analysis, filtering, and visualization are integrated in real time reducing time-to-insight.  Users can integrate CARTO’s API’s and geocoding services to complement other apps and business applications and can be integrated with custom proprietary analytical models.  CARTO can be used as an engine to visualize a wide range of data services.

CARTO is scalable and offers a Software as a Service  (SaaS) deployment model to push new features instantly allowing users to “grow as you go.” Being enterprise-ready also means making on-premise and private clouds architecture solutions available to clients.  CARTO also offers a mobile platform.

Sample Products and Applications

On October 16, 2019, CARTO hosted the 2019 Spatial Data Science Conference (SDSC) at Columbia University which I covered and reported on in a previous blog post.  Typically GeoSpatial Business Spotlight focuses on three or four applications from the firm being highlighted.  However, since SDSC was a day-long series of entirely CARTO-based applications, the conference website provides a better and more thorough overview on how CARTO is applied in business, academia, government, and nonprofit organizations.  Choose from presentations by Uber, Facebook, University of Chicago, American Securities, Salesforce Maps, and MIT among others.  In Empire State, CARTO supports numerous programs in the metropolitan New York City area in both business and government.

Contributions to the Profession 

As part of CARTO’s long-standing commitment to FOSS, Open Source, Open Data, and Open Science, the company has collaborated with many organizations providing access to next generation geospatial technology, data, and models. Most recently (October 2019), CARTO’s Javier de la Torre  joined the Urban Computing Foundation (UCF) Technical Advisory Committee which is a neutral forum for accelerating geospatial open source and community development.  The UCF operates under the umbrella of The Linux Foundation.  In July 2019, Geospatial Media and Communications included Javier de la Torre as part of the Location Analytics & Business Intelligence (LA & BI) Advisory Board.  Additional

CARTO is an open source software built on PostGIS and PostgreSQL which was first released in Beta at FOSS4G in September 2011 and officially released at Where2.0 in April 2012.  The CARTO software solution uses JavaScript extensively in front end web applications, back end Node.js based APIs, and for client libraries.

Overall, CARTO’s platform consists of the following primary components:

The CARTO platform enables users to access and manage vast amounts of data while at the same time providing numerous processes to discover, filter, and integrate local and Big Data libraries.  Geo-enabling large datasets provides a means to visualize and better understand large and complex datasets. CARTO enriches user location data with versatile, relevant datasets, such as demographics and census, and advanced algorithms, drawn from CARTO’s own Data Observatory and offered as Data as a Service (DaaS).

CARTO uses widget-driven dashboards, an array of maps, and unified workflows so that non-GIS and non-mapping users/staff can bring the power of location into the organization’s decision making.

The CARTO software user interface provides both user-friendly mapping and dashboard visuals which can be customized to user needs and experience.

Complex analysis, filtering, and visualization are integrated in real time reducing time-to-insight.  Users can integrate CARTO’s API’s and geocoding services to complement other apps and business applications and can be integrated with custom proprietary analytical models.  CARTO can be used as an engine to visualize a wide range of data services.

CARTO is scalable and offers a Software as a Service  (SaaS) deployment model to push new features instantly allowing users to “grow as you go.” Being enterprise-ready also means making on-premise and private clouds architecture solutions available to clients.  CARTO also offers a mobile platform.

Contact:

Florence Broderick
VP Marketing
flo@carto.com
4475-686-89402

 

Spatial Data Science Conference 2019

I had the opportunity to attend the 2019 Spatial Data Science Conference (SDSC) at Columbia University on October 16th.  Hosted by NYC-based Carto, the event was attended by over 500 people from government, nonprofits, industry and business, and academia.  The day-long conference was highlighted by a variety of presentations and lightning talks from data scientists and program managers representing organizations from around the world including Uber, Airbnb, Datarobot, Waze, Instacart, MIT, Sidewalk Labs, Two Sigma, and Facebook among others.  All of the presenters use Carto’s Software as a Service (SaaS) platform which provides GIS functionality, web mapping, and spatial data science tools.

Data Science

Founded in 2017, SDSC  brings together organizations who are pushing the boundaries of spatial data modelling – ranging from large enterprise, to cities and government, as well as thought leaders from academic institutions.  Shown through the use of geospatial applications and organizational project initiatives, there was no doubt the common denominator and focus of those attending:  Data. It was definitely worth the trip.  SDSC is similar, but a very different kind of a “geospatial conference” for those of us who have spent a career running in traditional government geospatial circles.  Attendees and presenters are largely a completely different make-up from those normally attending the annual New York State GIS conferences.  (Of the 500+ preconference registrants, nearly 55% were from the private sector).   The day’s event included high quality presentations on the latest in modelling techniques, data science and analytics, visualization practices, and new data streams.  This later issue an increasingly important and interesting one across the statewide geospatial community as the day’s conversation clearly illustrated both the growing number of online geospatial data sources  (for example, numerous references were made to U.S. Census Bureau American Community Survey) and related data access tools.  Speakers noted both the importance and benefits of open data portals though not without the caveat that it was not uncommon to have to clean and often normalize the data prior to using in applications.  Numerous references to popular sites such as GitHub, Leaflet, OpenStreetMap, and Elastic (which presented at the event) were made with regard to supporting the open source ecosystem.

Mudit Srivastav from Australia-based Origin Energy, presented on the use of spatial data to support the increase sales of residential roof top solar panels. Note the many types of data the company is using to better define growth areas.

Data for the Social Good

Another common theme throughout the day was the use of geospatial data for the social good.  Interesting to hear the point being made not only from nonprofits and academia, but from the private sector as well.  Way far away from the normal Albany GIS crowd, Stuart Lynn made a presentation focusing on how Two Sigma, an investment management firm located in New York City, provides spatial analysis support through the company’s Data Clinic program to nonprofits, academic institutions, and government organizations.  Their focus:  Enabling and promoting social impact through data driven predictive models while funding breakthrough research, education and a wide range of charities and foundations.    The image below identifies some of their recent projects.  Great stuff and I’m already planning on a more in-depth article with Stuart in the future.

Enabling Social Impact Organizations with Spatial Analysis Techniques

Stuart’s talk, as well as others during the day, made reference to the  The Data for Good movement which was a social media movement first started by DataKind to highlight how data science could be used to help address a range of humanitarian issues. DataKind is a collaborative network of organizations that work together with data scientists to leverage the science of data for social impact.  DataKind’s afternoon presentation “Spatial Data Science for Social Good:  Improving Access to Dignified Sanitation in Haiti” was excellent.  The video for this session and all of the day’s presentations are now available online at the Spatial Data Science Conference website.

Arguably, the demand for “data scientists” will increasingly intersect and expand in government geospatial programs in areas such as the health and human services, climate change, public safety, sustainability and resiliency,  and social and environmental justice to name only a few.   As the universe of geospatial data continues to expand and be integrated with both new artificial intelligence (AI) and machine reading technologies, as well as combined with the availability of more powerful GIS software, it is easy to see where the demand for data scientists focusing on locational and predictive analytics is headed.

I asked Javier de la Torre, Carto founder and SDSC organizer, what his impressions were of the day’s presentations and content.  He replied:

“First, moving the data discussion from WHERE to Why. Time to move to analyzing data using maps opposed to just seeing data in maps.  Second, the rise of the Spatial Data Scientist and/or where advance GIS is emerging as a new platform. And third, identifying the need for better data marketplaces which provides interactive solutions resulting in increased performance to users”

Links to the 2017 and 2018 presentations are also available on the SDSC website.

Mobility for All Abilities Hackathon

While I personally did not attend the September 21st Mobility for All Abilities Hackathon at the  Brooklyn Army Terminal, I had originally intended to put together a short summary article to post on the day’s events and activities after gathering information from BetaNYC staff.  But instead, check out BetaNYC’s own detailed summary here – its quite good.

The event coincided with the 2019 National Day of Civic Hacking and was co-hosted by by BetaNYC and MakerSpace NYC which partnered with  NYC Department of Transportation, NYC Mayor’s Office of People with Disabilities, NYC Mayor’s Office of Data Analytics, and other organizations to setup eight challenges for a day-long map-a-thon and civic hacking to improve mobility equity in New York City.  Incredible results!

New York City Council Member Brad Lander provided opening remarks at the Mobility for All Abilities Hackathon. Photograph by Charlie Hartwell / CC BY

I’ve written about BetaNYC before and they have presented at the Westchester GIS User Group meeting in the past. They fill a much needed and recognized space in metro NYC towards facilitating these types of technology/geospatial events which can serve as a model for similar civic focused programs across the state.

For more information on their programs and activities, visit the BetaNYC website.

 

The Geography of an Adirondack High Peaks Unsupported Thru-Hike Record

Background

Since early July this summer when our youngest son Lukas and close friend Mike Jock completed and established  a new record for an unsupported thru-hike of the 46 Adirondack High Peaks, I’ve been asked by various friends and colleagues across the region (who became aware of the accomplishment) as to whether or not any professional geospatial/GIS technology was used in some capacity in supporting the effort.   Each time my humbling answer has been pretty much the same:  Not Really.  However, they did carry some new gadgetry which is currently used ubiquitously across the adventure and extreme sports landscape to collect data that was used to authenticate their achievement.  But more on that later.

Starting their trek at the trail head to Seward Mountain at 3:30 AM Thursday, June 27th on the west side of the Adirondacks, the two covered the 46 High Peaks in 7 days, 2 hours, and 52 minutes finishing by summitting Mount Esther, on the northern shoulder of Whiteface shortly after 3 AM on July 4th. Ultimately making it down to the Mount Ester trail head at 6:20 AM to complete the trip.   All said, nearly 212 miles and 70,000 feet gained in elevation – more than twice the elevation of Mt. Everest.  They averaged well over a marathon a day, covering some 25-30 miles. On the final day they kept moving for more than 23 hours covering 45 miles.

Michael, left, and Lukas at the summit of Armstrong Mountain with Gothics Mountain in the background. The two were soccer teammates at SUNY Cortland. Mike King photo.

Preparation was more old school than one would think.  Nothing in this space replaces on-the-ground knowledge of the trails, trail junctions, where and when to bushwhack, locations of rock cairns for reference points, as well as planning and hiking/testing certain stretches of the route ahead of time.  Yeah, practice runs.  Luke had the benefit of many years of experience having first started ADK trail hiking since he was barely in kindergarten.  Like many other young children, his first “mountain” being Mt. Jo overlooking Heart Lake adjacent Adirondack Loj.  Safe to assume that through the family ties, his spatial orientation has always been pretty decent. Mike, a local boy, grew up in the Plattsburgh area.

This was their third attempt.   And such accomplishments don’t come without the element of good fortune avoiding injuries (which was the case in 2018) and a bunch of days with generally decent weather.  The 2018 attempt was also impacted due to an encounter with a black bear.  Yet another ADK backcountry hiker lesson in overnight food storage.

Given the enormity, location, and challenges of the effort, it would clearly not be safe, if not irresponsible,  to rely solely on the new generation of lightweight/handheld navigational devices as a means to lead one through the Adirondack Mountain wilderness.  There is no substitute for first-hand knowledge of the trail network and a pre-determined plan.  They carried only the essentials such as lighweight sleeping bags, tents, change of clothing, raingear, first aid/basic emergency gear, water filters,  and food.  Counting every ounce in keeping their backpacks to about 40 pounds.  Even omitting the weight of gas canisters/gas stove and in doing so did not prepare anything hot the entire trip.  Zero assistance.

Cool Gadgets

It’s amazing how far the adventure mapping and video/data collection market has evolved since GoPro technology was introduced nearly a decade ago, if not earlier.  The ability to playback Mike and Luke’s accomplishment – made available below via a series of links – is amazingly made possible by just a one simple mobile (wearable) component.  They used a Garmin Fenix 5 GPS Watch to collect X,Y and elevation data which, based on wireless connectivity, automatically uploads to Garmin Connect.  Upon completion of their hike, the entire trip dataset was downloaded to Relive an outdoor 3D video website which allows further rendering and interactive use of the data.    Two Anker PowerCore 26800 pre-charged power banks kept their phones and Garmin device charged.   Of sufficient size to recharge their phones/devices numerous times over.  They took two but in fact only needed one.   The Garmin app also collects heart rate, estimated calories burned, pace (minutes per mile), elevataion gain/loss, and total daily distance – all of which is made available in interactive and graphic form.  In addition to the default Google basemap, data can be rendered on other base maps including satellite, terrain, OpenStreetMap, and Here.  Amazingly compact, yet powerful tools.

The Daily Numbers

The following section includes a brief narrative – Mike’s own words- and links which can be used to highlight trail and body metrics as well as mapping visuals for each hiking day.  For those familiar with hiking and trekking in the Adirondacks, the visuals certainly help put Luke and Mike’s daily segments in better perspective.

Hold on to your seats.

Thursday June 27

Hiker Notes: 3:30 am depart from Corey’s Road.   Day consisted of seven mountains: Seward, Donaldson, Emmons, Seymour, Santanoni, Panther & Couchsachraga. Out and backed the first four.  Trekked approximately 10 miles to Santanoni range.  Set up camp at the base, then we went up the expressway to summit Santanoni first, followed by Panther, and then finished with Couchsachraga. We came down the Panther brook trail before staying at camp at base of expressway again. Approximately 37.84 miles.

Instructions for all similar images below: Click on map for additional graphics and daily metrics. Or click the black triangle button at the bottom of the screen to see hiker movement over the course of the day.

Instructions for all similar Relive 3D Interactive links below: Click on the icon circled in red to see daily hiker movement on top of a 3D base map and associated photos. Clicking on the “View Interactive Route” button enables the ability to interact with the map and trail data “twirling” the image to see from the north, east, south, and west perspective.

Friday June 28

Hiker Notes: From camp, hiked over to Allen mountain, out and backed it before moving up towards Colden/Avalanche Lakes. We out and backed Mount Marshall before setting up camp over the dam at Lake Colden. We summited two mountains and covered approximately 28.47 miles.

June 28thRelive 3D Interactive

Saturday June 29

Hiker Notes: The ADK Loj Loop day. Summitting eight mountains and doing a big circle before returning to the same place we camped the night before. We started by going up the backside of Colden, then Tabletop, Phelps, Street, Nye, Wright, Algonquin and finishing on Iroquis. Approximately 32.18 miles

Note: The only day “noise” was either user generated or introduced by the Garmin device itself. In the vicinity of Street and Nye – Illustrated by the “straight” additional red line. Also reflected in the Relive 3D viewer.

June 29thRelive 3D Interactive

Sunday June 30

Hiker Notes:   Audible day. Started by moving to the base of Cliff and Redfield (out and bacs), then trekking up towards Gray, Skylight, Marcy, Haystack, Basin and finishing with Saddleback. Due to it raining all day, made the game time decision to check out and head towards Johns Brook Lodge where we camped at the trailhead of Big Slide Mountain (setting ourselves up the next day to finish the Great Range and start moving east). Summitted eight mountains and covered approximately 22.5 miles.

June 30th:  Relive 3D Interactive

Monday July 1

Hiker Notes:  Out and backed Big Slide early in the AM, before climbing up towards Great Range. First we summited Lower Wolf Jaw, then Upper Wolf Jaw, Armstrong, Gothics and Sawteeth. We then headed down trail to Lower Ausable Lake. Next we did Colvin and Blake before camping between those two mountains.   In total we summited eight peaks again and covered 23.97 miles.

July 1st:  Relive 3D Interactive

Tuesday July 2

Hiker Notes: Started by ascending Nippletop, moved on over to Dial, then bushwhacked down towards the Dix Range. Once we got down from bushwhack/brook we dropped bags and then out and backed the entire Dix Range: Dix, Hough, South Dix, Grace Peak & Macomb. From there we got our bags again and then moved our stuff over to Roaring Brook and the base of Giant Mountain. In total we covered 25.26 miles and summitted seven peaks.

July 2nd:  Relive 3D Interactive

 Wednesday July 3

Hiker Notes:  The 24 hour day. Woke up at 3 am and went to get Giant and Rocky Ridge Peak (out and backs). From there we picked up our stuff again and schlepped ourselves 5.5 miles down route 73 toward Marcy airfield. Next up was cutting from one side of 73 to the Lake Placid side of 73 along the ridge trail that summitted Porter and Cascade Mountains. Off of those we had then a 10 mile stretch on route 73 and then River Road towards the trai lhead to Whiteface mountain. Using the trail by Connery pond we entered that portion at 1030 PM and was able to summit both Whiteface and then Esther by 3:15 am. About 23 hour and 30 minutes of moving time we summitted our final 6 peaks and covered 44.18 miles. Passed out for an hour and reached Memorial Highway by 06:22 AM, where a time-stamped photo was taken.

July 3nd:  Relive 3D Interactive (Part 1)  Incredible day!!

July 4th:  Relive 3D Interactive (Part 2)

Summary

Quite an effort  and on top of setting the record, between this trip and last year’s attempt, the two helped raise nearly $18,000 for the Dubin Breast Center at Mount Sinai Hospital in New York.  Their accomplishment was covered by Backpacker Magazine as well as a selected number of upstate New York online publications (UpstateNewYork and NNY360) among others.  The articles are generally similar in content though do offer more details about food and other daily nuances/items which I omitted here.  Choosing rather to highlight and focus on the immense geographic element of the effort.

Both Luke and Mike are personal trainers living and working out of New York City.  More information can be found on their website Team Adventure Portal.