We are rebuilding our entire GIS analysis platform and are planning to use the esri managed data store for user generated content that is not in official support.
I understand it is a Postgres black box that is managed via esri applications and a few .bat files on the server.
I've used the Portal reader in FME to access some of the features but found it slow.
Is anyone using FME to connect directly to the Postgres database to access the features in the managed data store?
I'm curious where you have gotten on this and what others think? My understanding from the research I have done is that ESRI locks up this ArcGIS Data Store Relational Postgres database as tightly as humanly possible, in order to force you to integrate with it through the ArcServer Hosted Services. As data professionals we should have a real problem with that pattern, and the way it limits the ability to interact with that data. I would encourage anyone to not implement the hosted services as your only pattern, it may have a place in the right situation, but for data you want to integrate with other systems, the services leveraging data from a traditional Enterprise Geodatabase pattern is what I would recommend, though that does limit what you can use out of the ESRI stack against those services.
I'm curious where you have gotten on this and what others think? My understanding from the research I have done is that ESRI locks up this ArcGIS Data Store Relational Postgres database as tightly as humanly possible, in order to force you to integrate with it through the ArcServer Hosted Services. As data professionals we should have a real problem with that pattern, and the way it limits the ability to interact with that data. I would encourage anyone to not implement the hosted services as your only pattern, it may have a place in the right situation, but for data you want to integrate with other systems, the services leveraging data from a traditional Enterprise Geodatabase pattern is what I would recommend, though that does limit what you can use out of the ESRI stack against those services.
Our plan is to put maintained data in SDE and let business users use the data store for analysis output from Portal for ArcGIS. Also spatio temporal data is destined for the data store and I hear that raster can go there too.
If data is to be supported it will go into SDE but this will be very hard to control. My thoughts are that if a user in the business was to leave and his/her account was deleted (and the analysis data in the data store) and it wouldn't make a difference then it could go in the data store. Or the content from the person leaving must be associated with another account, which the business will be responsible for organising.
We are busy building a new platform so at the early stages. After a few weeks I already see lots of test services connected to lots of test data in this data store so I am not so positive about it. It's hard to know what services/data are to be supported by the GIS admin team among all the junk.
Our AGS Administrator has shown me the Postgres interface on the 10.5.1 AGS box but I haven't had time to test. Also the idea is that you don't connect directly to it via Postgres. I have successfully made a connection to the datastore via a Portal/AGOL feature service reader.
Some articles worth reading:
ArcGIS Data Store and RDBMS?
https://community.esri.com/thread/191678-arcgis-data-store-and-rdbms
ArcGIS Data Store Under the Hood
https://community.esri.com/thread/160435
‘The
Datastore, in my humble opinon, is meant to be a wild-wild-west of sorts. That
is, since you don't want users creating, editing and deleting authoritative
data but you also want them to be able to do some self-servicing of their basic
GIS needs, you need a place for them to host their own data. That's the point
of the datastore. It's a database that you can set up and pretty much not have
to worry about. It's a decentralized database to host all the crazy
permutations of end user data. The data within the datastore is not
authoritative - it's user data. All their little files and personal layers.
When it comes to authoritative content -
which where I work, we tend to define as content that a team of GIS
Professionals creates, validates and publishes for end users to leverage
within the Portal - that's where setting up an Enterprise Geodatabase and
registering it with your federated server(s) comes into play. To do this, you
set up a normal Enterprise Geodatabase on SQL Server/Oracle/PostgreSQL/etc. -
whatever your DB technology of choice is and then register that enterprise
geodatabase with each of your federated servers.
Once that's done, you load your
authoritative content into that enterprise geodatabase and work with it the
same way you always have. Direct-Connect from Catalog and run GP tools against
it or if you're brave, work with it at the DB level. The key to making this
work seamlessly with Portal is federation. You must federate your Portal with
ArcGIS for Server. Doing so means that anything you publish to ArcGIS for
Server will be made available in Portal. It also means that your Portal Groups
and Roles will determine who has access to the services being sourced from your
Enterprise Geodatabase.’
I'm curious where you have gotten on this and what others think? My understanding from the research I have done is that ESRI locks up this ArcGIS Data Store Relational Postgres database as tightly as humanly possible, in order to force you to integrate with it through the ArcServer Hosted Services. As data professionals we should have a real problem with that pattern, and the way it limits the ability to interact with that data. I would encourage anyone to not implement the hosted services as your only pattern, it may have a place in the right situation, but for data you want to integrate with other systems, the services leveraging data from a traditional Enterprise Geodatabase pattern is what I would recommend, though that does limit what you can use out of the ESRI stack against those services.
Our plan is to put maintained data in SDE and let business users use the data store for analysis output from Portal for ArcGIS. Also spatio temporal data is destined for the data store and I hear that raster can go there too.
If data is to be supported it will go into SDE but this will be very hard to control. My thoughts are that if a user in the business was to leave and his/her account was deleted (and the analysis data in the data store) and it wouldn't make a difference then it could go in the data store. Or the content from the person leaving must be associated with another account, which the business will be responsible for organising.
We are busy building a new platform so at the early stages. After a few weeks I already see lots of test services connected to lots of test data in this data store so I am not so positive about it. It's hard to know what services/data are to be supported by the GIS admin team among all the junk.
Our AGS Administrator has shown me the Postgres interface on the 10.5.1 AGS box but I haven't had time to test. Also the idea is that you don't connect directly to it via Postgres. I have successfully made a connection to the datastore via a Portal/AGOL feature service reader.
Some articles worth reading:
ArcGIS Data Store and RDBMS?
https://community.esri.com/thread/191678-arcgis-data-store-and-rdbms
ArcGIS Data Store Under the Hood
https://community.esri.com/thread/160435
‘The
Datastore, in my humble opinon, is meant to be a wild-wild-west of sorts. That
is, since you don't want users creating, editing and deleting authoritative
data but you also want them to be able to do some self-servicing of their basic
GIS needs, you need a place for them to host their own data. That's the point
of the datastore. It's a database that you can set up and pretty much not have
to worry about. It's a decentralized database to host all the crazy
permutations of end user data. The data within the datastore is not
authoritative - it's user data. All their little files and personal layers.
When it comes to authoritative content -
which where I work, we tend to define as content that a team of GIS
Professionals creates, validates and publishes for end users to leverage
within the Portal - that's where setting up an Enterprise Geodatabase and
registering it with your federated server(s) comes into play. To do this, you
set up a normal Enterprise Geodatabase on SQL Server/Oracle/PostgreSQL/etc. -
whatever your DB technology of choice is and then register that enterprise
geodatabase with each of your federated servers.
Once that's done, you load your
authoritative content into that enterprise geodatabase and work with it the
same way you always have. Direct-Connect from Catalog and run GP tools against
it or if you're brave, work with it at the DB level. The key to making this
work seamlessly with Portal is federation. You must federate your Portal with
ArcGIS for Server. Doing so means that anything you publish to ArcGIS for
Server will be made available in Portal. It also means that your Portal Groups
and Roles will determine who has access to the services being sourced from your
Enterprise Geodatabase.’