Hi,
I dont use this format often, but it seems to me that just reading both/all sources into a final result will "update" the files.
Itay
The change info file contains information about deletions as well as updates so it's not as simple as putting together the data from the full extract and the change only.
Thank you both for your replies. As EGomm says, mastermap COU gz file contains additions and removals of features.
(I have been testing this using the TopogrphicLine part of mastermap)
Firstly i tried using my current mastermap file (.tab) and the relevant section of the COU file (.gz) as inputs going to the same output. This added both the datasets together (I do not think that lines were duplicated which was good). It added in the new additional lines but unfortunately did not do the removals.
Out of curiosity I added the 'DepartedFeature' part of the gz file also as an input but this just added the polygons (bounding boxes) of the DepartedFeatures into the final file.
I was hoping that I might find someone who had done this before as I am sure it must possible to do but there is very little information about processing mastermap online. It would be very useful to process mastermap through FME, otherwise we would have to invest in other software (mastermap loaders)
For smaller geographic areas, it's probably as easy just to take a full supply each time and process that rather than a change only update.
I think Jonathan Moules at Warwick County Council processes mastermap using FME so it may be worth contacting him?
Further to my last post I am considering trying various options:
I am going to attempt to use the clipper feature on the current file with the DepartedFeatures as the bounding boxes (hopefully there is a setting to say clip only those completely within and also that there arent other random features completely in the bounding box)
Alternatively the DepartedFeatures file contains a list of TOIDS (the unique ID for each feature in mastermap) and it is possible there is a transformer that can utilise this to remove the features.
Thanks EGomm, I will contact him
I agree, for a small area resupply makes sense, but this is the whole county to be used across the council by many users with updates every 2 months so the COUs are needed
If you want to have a go yourself, a couple of important points to remember are
one departed feature record can cover features in a number of themes.
modified features need to be replaced rather than inserted
I did wonder what happened about modified features. I assumed that they were deleted and readded (is that not the case?)
When you say "a number of themes" do you mean that somemay relate to e.g.TopLine but some may relate to e.g. CartoSymbol?
I am mainly working by trial and error at the moment. Despite that fact that I am having no luck at the moment in getting it to work at least I am getting lots of practice at using FME (I am quite new to it)
The clipper didnt help because it deleted everything intersecting the bounding box rather than features which were completely within it
As you appear to be writing to TAB, a format that doesn't allow you to UPDATE it. You'll need to do some comparisions of your CoU against your processed Initial supply and then write out a new amalgamated TAB holding but this can get tricky without a database.
It's more helpful if you can consider using a database as your starting point and then this is the environment from which you publish your MasterMap data. Similar to:
- Initial supply GZ to database (perhaps PostGIS). Use a DuplicateRemover on the way in as OSMM contains duplicates between chunks.
- Load CoU to database, comparing each feature.
- If Toid exists but 'version' is incremented - UPDATE
- If Toid does not exist - INSERT
- You can then continually apply CoU to the master holding and the have another workspace to read from the database to produce your TABs when required.
I agree with @1spatialdave on this. We hold mastermap data and COU updates are quite involved and tricky. I think going down the DB route and not using flat files is the best way. It will save you lots of grief in the long run as well as making back ups a lot easier not to mention speed. Its not unusual for a COU to update hundreds of thousands of rows so writing that out to flat files will be slow.
Personally we use FME to convert and write into sql and then use SQL scripts and stored procedures to do all the updates, we found it quicker and easier to do this way. The stored procedures can be set up to automatically back up any features that have been changed, deleted or updated and therefore you have a separate mastermap holding of all the old data which can be useful.
Mapinfo can read in the data straight from SQL as well negating the need for tabs alltogether.
