How do I scrape data from an ArcGIS Online map?

Question

I want to scrape the data from an ArcGIS map. The following map has a popup when we click the red features. How do I access that data programmatically?

Link : https://cslt.maps.arcgis.com/apps/MapSeries/index.html?appid=2c9f3e737cbf4f6faf2eb956fa26cdc5

Emblements

Answer 1

Note: Please respect the access and use constraints of any ArcGIS Online item you access. When in doubt, don't save a copy of someone else's data.

The ArcGIS Online REST interface makes it relatively simple to get the data behind ArcGIS Online items. You need to use an environment that can make HTTP requests and parse JSON text. Most current programming languages either have these capabilities built in or have libraries available with these capabilities.

Here's a general workflow that your code could follow.

1. Use the app ID and the item data endpoint to see the app's JSON text:

   https://www.arcgis.com/sharing/rest/content/items/2c9f3e737cbf4f6faf2eb956fa26cdc5/data

2. Search that text for webmap and see that the app uses the following web maps:

   • d2b4a98c39fd4587b99ac0878c420125
   • 7b1af1752c3a430184fbf7a530b5ec65
   • c6e9d07e4c2749e4bfe23999778a3153

3. Look at the item data endpoint for any of those web maps:

   https://www.arcgis.com/sharing/rest/content/items/d2b4a98c39fd4587b99ac0878c420125/data

4. The list of operationalLayers specifies the feature layer URLs from which you could harvest data. For example:

   https://services2.arcgis.com/gWRYLIS16mKUskSO/arcgis/rest/services/VHR_Areas/FeatureServer/0

5. Then just run a query with a where of 0=0 (or whatever you want) and an outFields of *:

   https://services2.arcgis.com/gWRYLIS16mKUskSO/arcgis/rest/services/VHR_Areas/FeatureServer/0/query?where=0%3D0&outFields=%2A&f=json

   Use f=html instead if you want to see a human-readable request form and results.

   Note that feature services have a limit of how many features you can get per request, so you will probably want to filter by geometry or attribute values. Read the documentation to learn everything you can do with feature service queries.

Socorrosocotra

yes! i got your point regarding the data rights! Data was required for educational purposes with the permission of the city lake authorities! and thanks alot your answer did helped alot. – Emblements

I'm glad my answer helped. – Socorrosocotra

Is this answer still valid? I was looking for data sources here. Found no webmap, just mapWidget, then datasets.dataSource.id – which looks like this: 1200f3f1-8f72-4ea6-af16-14f19e9a4517 – it doesn't seem to work with /sharing/rest/content/items/<itemId>/data requests – Coloquintida

Yes, this answer is valid. The item you're looking at is a dashboard, which is a different structure than the story map that the OP asked about. In the dashboard item data, there are three mapWidget objects, each with an itemId that is a web map item ID. You can use any of those items' data (here, for example; the other two seem almost identical to me) and start at step 3 in this answer. – Socorrosocotra

This was incredibly helpful! – Antofagasta

That's great that this answer is helping people, even if it was not marked as the accepted answer yet. 😃 – Socorrosocotra

is it possible to scrape the data of a map like this? - sig.sea.gob.cl/mapaLineasBaseEIA - it doesn't have an arcgis.com url, thanks – Swollen

I'm having some problems to retrieve data from a dashboard but this is REALLY helpful – Cedric