Dissolve issues with very large merged landcover dataset

5275
19
07-18-2016 01:29 PM
JosephWalderman1
New Contributor II

I have taken a large mosaic land cover dataset for the James River Watershed and manipulated it through iterate, clip, and raster to polygon then merging all the resulting shapefiles into a file geodatabase. The resulting merged file is quite large, 8GB and 22 million features. I tried to run the dissolve with ArcMap 10.4.1 but received invalid topology or topoengine errors, sometimes with (out of memory), even though I am running a pc with 12GB RAM. I am trying to run it on ArcGIS Pro since that automatically utilizes my computers 64-bit geoprocessing capability and it worked well for a while, getting up to 87% in about 12 hours but it has been stuck at 87 for the past 8 hours. I really do not want to cancel and have all this time wasted. Any recommendations?

0Kudos
19 Replies
JosephWalderman1
New Contributor II

Only other thing I can think of it to try multi part but I feel like that would mess with the lines especially with such high-res data

0Kudos
ChrisDonohue__GISP
MVP频繁的贡献者

Some ideas:

    • Maybe take an approach of merging the data into say 8 "chunks", dissolving each, then merging all 8 and dissolving again.
    • Shapefiles do not store parametric curves, File Geodatabases do, so going from Shapefiles to File Geodatabase can lead to many slivers generated, which can then lead to quite a bit of topology cleanup.
    • ESRI Help article - Tiled processing of large datasetsArcGIS Help (10.2, 10.2.1, and 10.2.2)

克里斯不ohue, GISP

JosephWalderman1
New Contributor II

I am not exactly sure how to partition data like that, how would that work?

0Kudos
DanPatterson_Retired
MVP Esteemed Contributor

do the merge in Pr in a geodatabase, a shapefie can't be > 2 GB so you are fighting a losing battle there if you have to have everything in one layer and can't partition. speaking of partition, you could spatially tile your data and run it on the bits, but I would consider either partition spatially or separate your attributes and rejoin llater.. I don't know if woring on non-locally saved data would improve things. What sort of spatial resolution are you using with the raster to polygon? and is it needed, given the data were in raster format to begin with. I assume that you didn't try to make a finer resolution than existed in the raster.

DarrenWiens2
MVP Honored Contributor

Actuuuually .shp + .dbf可以超过2 gb(但neither individually). Just a fun fact:FAQ: Are there file size limitations for shapefiles?

Anyhow, since some of your errors seem to indicate geometry issues, you may want to runCheck GeometryorRepair Geometryon your feature class.

DanPatterson_Retired
MVP Esteemed Contributor

I always forget, since I rarely have attributes other than FID, Shape and ID ... my bad for those in attribute world and not in geometry world

0Kudos
JosephWalderman1
New Contributor II

So does it seem like it would be best to just abandon the dissolve at 87%?

0Kudos
DarrenWiens2
MVP Honored Contributor

I suppose you've considered staying in raster-world until you absolutely have to switch to vector? "Raster is faster" (although vector is correcter...)

JosephWalderman1
New Contributor II

I'm not exactly sure how I would be able to stay in raster world. I am trying to calculate core habitat so I need to dissolve by land cover type at some point and I am not sure I can get there by raster.

0Kudos