Skip to main content
Question

Mastermap Change Only Update (COU) in FME

A
Forum|alt.badge.img
I recently got a new cut of mastermap and converted it easily from the .gz files to the various .tab files (topo line, topo area etc). I was really pleased at how easy this was to achieve as the 'generate workspace' built the whole workspace for me.

 

 

This week I got my first COU disk but cannot work out how to apply this update to my current .tab files. At first I tried just running the conversion to my current files as the output but this just overwrote the files. (glad i took a backup)

 

 

Are there some settings you need to tweek to append the files rather than overwrite? or do you need to process the data after it has been converted from gz to tab (although I assumed that the gz files contained invisible instructions that explains what to add and what to delete)?

 

 

Can anyone help me with this please because I assume it must be possible using FME. I have been playing around with different transformers (such as ChangeDetector) but am having no luck

10 replies

fmelizard
Safer
Forum|alt.badge.img+18
  • fmelizardSafer
  • Safer
  • January 8, 2014
Hi,

 

 

I dont use this format often, but it seems to me that just reading both/all sources into a final result will "update" the files.

 

 

Itay
ebygomm
Influencer
Forum|alt.badge.img+38
  • ebygommInfluencer
  • Influencer
  • January 8, 2014
The change info file contains information about deletions as well as updates so it's not as simple as putting together the data from the full extract and the change only.
A
Forum|alt.badge.img
  • alison
  • Author
  • January 9, 2014
Thank you both for your replies. As EGomm says, mastermap COU gz file contains additions and removals of features.

 

 

(I have been testing this using the TopogrphicLine part of mastermap)

 

 

Firstly i tried using my current mastermap file (.tab) and the relevant section of the COU file (.gz) as inputs going to the same output. This added both the datasets together (I do not think that lines were duplicated which was good). It added in the new additional lines but unfortunately did not do the removals.

 

 

Out of curiosity I added the 'DepartedFeature' part of the gz file also as an input but this just added the polygons (bounding boxes) of the DepartedFeatures into the final file.

 

 

I was hoping that I might find someone who had done this before as I am sure it must possible to do but there is very little information about processing mastermap online. It would be very useful to process mastermap through FME, otherwise we would have to invest in other software (mastermap loaders)
ebygomm
Influencer
Forum|alt.badge.img+38
  • ebygommInfluencer
  • Influencer
  • January 9, 2014
For smaller geographic areas, it's probably as easy just to take a full supply each time and process that rather than a change only update.

 

 

I think Jonathan Moules at Warwick County Council processes mastermap using FME so it may be worth contacting him?
A
Forum|alt.badge.img
  • alison
  • Author
  • January 9, 2014
Further to my last post I am considering trying various options:

 

 

I am going to attempt to use the clipper feature on the current file with the DepartedFeatures as the bounding boxes (hopefully there is a setting to say clip only those completely within and also that there arent other random features completely in the bounding box)

 

 

Alternatively the DepartedFeatures file contains a list of TOIDS (the unique ID for each feature in mastermap) and it is possible there is a transformer that can utilise this to remove the features.
A
Forum|alt.badge.img
  • alison
  • Author
  • January 9, 2014
Thanks EGomm, I will contact him

 

 

I agree, for a small area resupply makes sense, but this is the whole county to be used across the council by many users with updates every 2 months so the COUs are needed
ebygomm
Influencer
Forum|alt.badge.img+38
  • ebygommInfluencer
  • Influencer
  • January 9, 2014
If you want to have a go yourself, a couple of important points to remember are 

 

 

one departed feature record can cover features in a number of themes.

 

modified features need to be replaced rather than inserted

 

A
Forum|alt.badge.img
  • alison
  • Author
  • January 9, 2014
I did wonder what happened about modified features. I assumed that they were deleted and readded (is that not the case?)

 

 

When you say "a number of themes" do you mean that somemay relate to e.g.TopLine but some may relate to e.g. CartoSymbol?

 

 

I am mainly working by trial and error at the moment. Despite that fact that I am having no luck at the moment in getting it to work at least I am getting lots of practice at using FME (I am quite new to it)

 

 

The clipper didnt help because it deleted everything intersecting the bounding box rather than features which were completely within it
davideagle
Contributor
Forum|alt.badge.img+21
  • davideagleContributor
  • Contributor
  • January 13, 2014
As you appear to be writing to TAB, a format that doesn't allow you to UPDATE it. You'll need to do some comparisions of your CoU against your processed Initial supply and then write out a new amalgamated TAB holding but this can get tricky without a database.

 

 

It's more helpful if you can consider using a database as your starting point and then this is the environment from which you publish your MasterMap data. Similar to:

 

  • Initial supply GZ to database (perhaps PostGIS). Use a DuplicateRemover on the way in as OSMM contains duplicates between chunks.
  • Load CoU to database, comparing each feature.
    • If Toid exists but 'version' is incremented - UPDATE
    • If Toid does not exist - INSERT
  • You can then continually apply CoU to the master holding and the have another workspace to read from the database to produce your TABs when required. 

 

M
Forum|alt.badge.img+5
  • marko
  • January 7, 2016

I agree with @1spatialdave on this. We hold mastermap data and COU updates are quite involved and tricky. I think going down the DB route and not using flat files is the best way. It will save you lots of grief in the long run as well as making back ups a lot easier not to mention speed. Its not unusual for a COU to update hundreds of thousands of rows so writing that out to flat files will be slow.

Personally we use FME to convert and write into sql and then use SQL scripts and stored procedures to do all the updates, we found it quicker and easier to do this way. The stored procedures can be set up to automatically back up any features that have been changed, deleted or updated and therefore you have a separate mastermap holding of all the old data which can be useful.

Mapinfo can read in the data straight from SQL as well negating the need for tabs alltogether.

Reply


Helpful Members This Week

Community Stats

31,331
Posts
118,863
Replies
38,957
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.