Skip to main content
  • EN

I have a Geospatial PDF with a number of layers in it which I am trying to export into shapefiles. The trouble is that the resulting shapefile have systematically misplaced the items, so I image that I must not be understanding something about how my Reader is reading the coordinate system. The source of the geoPDF is Canvas X GIS, which can export directly to a shapefile, expect that some of the information such as text objects are lost. I export the data into both shapefiles and the GeoPDF in NAD27 UTM 11N. When FME reads the GeoPDF, it assigns the coordinates as EPSG26711, which seems to be correct. I also tried manually assigning it as UTM27-11, although that seem to place the objects in the same place after running the workbench. However when I first read the layers, and then viewed the items within the Data Inspector, it did not seem to assign geographic coordinates. I think that it only did so after I ran the workbench. In the log, it talks about running reprojection engines for 'LL-WGS84' -> 'UTM27-11. I do not understand why it would need to do so as my data should not be in WGS84. Does anyone have an idea about why FME is misplacing my data?

Here is some of the log info that I get when I add another reader (to read the same layers and set the coordinate system to EPSG26711):

Predefined
coordinate system `UTM27-11' (UTM with NAD27 datum, Zone 11, Meter; Central
Meridian 117d W) matches dataset coordinate system

The OGC
definition of the FME coordinate system 'UTM27-11' is
'PROJCS["unnamed",GEOGCS["NAD27",DATUM["D_North_American_1927",SPHEROID["Clarke_1866",6378206.4,294.9786982139,AUTHORITY["EPSG","7008"]],AUTHORITY["EPSG","6267"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],DATUM_TRANSFORM[2,2],UNIT["degree",0.0174532925199433],AUTHORITY["EPSG","4267"]],PROJECTION["Transverse_Mercator"],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],PARAMETER["scale_factor",0.9996],PARAMETER["Central_Meridian",-117],PARAMETER["Latitude_Of_Origin",0],UNIT["Meter",1]]'

FME
Configuration: Using FME Reprojection Engine

CS-MAP
Reprojector: Transformation will be automatically selected for 'LL-WGS84' ->
'UTM27-11', and is not guaranteed to remain the same in future releases of FME

Reprojector:
Using transformation `NAD83_to_WGS84,Inverse(Null,EPSG:1188)' when reprojecting
from LL-WGS84 to UTM27-11

Reprojector:
Using transformation `NAD27_to_NAD83,Inverse(Grid File
Interpolation,EPSG:1241)' when reprojecting from LL-WGS84 to UTM27-11

And when I run the workbench it gives me this result:

Predefined
coordinate system `UTM27-11' (UTM with NAD27 datum, Zone 11, Meter; Central
Meridian 117d W) matches dataset coordinate system

The OGC
definition of the FME coordinate system 'UTM27-11' is
'PROJCS["unnamed",GEOGCS["NAD27",DATUM["D_North_American_1927",SPHEROID["Clarke_1866",6378206.4,294.9786982139,AUTHORITY["EPSG","7008"]],AUTHORITY["EPSG","6267"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],DATUM_TRANSFORM[2,2],UNIT["degree",0.0174532925199433],AUTHORITY["EPSG","4267"]],PROJECTION["Transverse_Mercator"],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],PARAMETER["scale_factor",0.9996],PARAMETER["Central_Meridian",-117],PARAMETER["Latitude_Of_Origin",0],UNIT["Meter",1]]'

FME
Configuration: Using FME Reprojection Engine

CS-MAP
Reprojector: Transformation will be automatically selected for 'LL-WGS84' ->
'UTM27-11', and is not guaranteed to remain the same in future releases of FME

Reprojector:
Using transformation `NAD83_to_WGS84,Inverse(Null,EPSG:1188)' when reprojecting
from LL-WGS84 to UTM27-11

Reprojector:
Using transformation `NAD27_to_NAD83,Inverse(Grid File
Interpolation,EPSG:1241)' when reprojecting from LL-WGS84 to UTM27-11

Reprojector:`NAD27_to_NAD83,Inverse(Grid
File Interpolation,EPSG:1241)' will use the following grid shift files:

Reprojector:
C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\conus.l?s

Reprojector:
C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\alaska.l?s

Reprojector:
C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\prvi.l?s

Reprojector:
C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\hawaii.l?s

Reprojector:
C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\stgeorge.l?s

Reprojector:
C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\stlrnc.l?s

Reprojector:
C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\stpaul.l?s

Reprojector:
If a point is outside the usable range of 'NAD27_to_NAD83' the following
fallback transformation will be used: 'NAD27-48_to_WGS84,Inverse(Multiple
Regression)'

NOT changing coordinate system of reader identified by
keyword `PDF2D_1' from `EPSG:26711' to `UTM27-11' -- mapping file setting of
`EPSG:26711' overrides coordinate system `UTM27-11' read from file

And when I run the workbench and reset the reader coordinate system to UTM27-11, it gives me slightly different results:

Predefined coordinate system `UTM27-11' (UTM with NAD27 datum, Zone 11, Meter; Central Meridian 117d W) matches dataset coordinate system

The OGC definition of the FME coordinate system 'UTM27-11' is 'PROJCS["unnamed",GEOGCS["NAD27",DATUM["D_North_American_1927",SPHEROID["Clarke_1866",6378206.4,294.9786982139,AUTHORITY["EPSG","7008"]],AUTHORITY["EPSG","6267"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],DATUM_TRANSFORM[2,2],UNIT["degree",0.0174532925199433],AUTHORITY["EPSG","4267"]],PROJECTION["Transverse_Mercator"],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],PARAMETER["scale_factor",0.9996],PARAMETER["Central_Meridian",-117],PARAMETER["Latitude_Of_Origin",0],UNIT["Meter",1]]'

FME Configuration: Using FME Reprojection Engine

CS-MAP Reprojector: Transformation will be automatically selected for 'LL-WGS84' -> 'UTM27-11', and is not guaranteed to remain the same in future releases of FME

Reprojector: Using transformation `NAD83_to_WGS84,Inverse(Null,EPSG:1188)' when reprojecting from LL-WGS84 to UTM27-11

Reprojector: Using transformation `NAD27_to_NAD83,Inverse(Grid File Interpolation,EPSG:1241)' when reprojecting from LL-WGS84 to UTM27-11

Reprojector:`NAD27_to_NAD83,Inverse(Grid File Interpolation,EPSG:1241)' will use the following grid shift files:

Reprojector: C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\conus.l?s

Reprojector: C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\alaska.l?s

Reprojector: C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\prvi.l?s

Reprojector: C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\hawaii.l?s

Reprojector: C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\stgeorge.l?s

Reprojector: C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\stlrnc.l?s

Reprojector: C:\\Program Files\\FME\\Reproject\\GridData\\Nadcon\\stpaul.l?s

Reprojector: If a point is outside the usable range of 'NAD27_to_NAD83' the following fallback transformation will be used: 'NAD27-48_to_WGS84,Inverse(Multiple Regression)'

Would you mind sharing your data (or a small part of it)? Also, I'd really like to know how far off the data is from where it should be.

If I understand you correctly you're only viewing it in the Data Inspector so far. With a background map? Because that would probably explain a lot of the reprojection messages. The data you view is reprojected on the fly to the system of the background map and this may involve a geoid transformation too (from NAD27 to WGS84 in this case)

Hi @alexandercaudet,

If you follow these steps, we may be able to better identify the issue you are experiencing:

  1. Create a new workspace to read your PDF file, and open up the reader settings while adding the reader.
  2. Change the reader setting "Coordinate Units" to "Page Points" (this will let us see things without trying to convert them to real-world coordinates).
  3. Enable the "Non-Spatial" group and set "Metadata Objects to Read" to "Pages" and "Map Frames".
  4. Log the features from the "pdf_page_metadata" and "pdf_frame_metadata" feature types. If you can route them to an Inspector and take a screenshot, that would be helpful too!

This will help us see the extents of each coordinate system area in your PDF document. I suspect that either:

  • Some features in your document are not within the extents of any map frame.
  • Some of your map frames overlap, and FME is picking the wrong one for features that are in the intersection area.

You may also find it helpful to inspect both your data and the "pdf_frame_metadata" feature types at the same time (in page-points mode), since this makes FME's decision process about coordinate systems a lot clearer.

@redgeographics Here is a link with a sample that I will keep valid until this post is solved.

https://drive.google.com/file/d/1YpDyCUZqvSgxxdrDvSacEsN4wfPw87ex/view?usp=sharing

I have been viewing it in Data Inspector but not with any background map, just the layers in the PDF which include both raster and vector data. I have also run the workspace and viewed the data within ArcMap where I have noticed the offset of 77m about east-southeast from what I presume the actual positions to be.

Hi @alexandercaudet,

If you follow these steps, we may be able to better identify the issue you are experiencing:

  1. Create a new workspace to read your PDF file, and open up the reader settings while adding the reader.
  2. Change the reader setting "Coordinate Units" to "Page Points" (this will let us see things without trying to convert them to real-world coordinates).
  3. Enable the "Non-Spatial" group and set "Metadata Objects to Read" to "Pages" and "Map Frames".
  4. Log the features from the "pdf_page_metadata" and "pdf_frame_metadata" feature types. If you can route them to an Inspector and take a screenshot, that would be helpful too!

This will help us see the extents of each coordinate system area in your PDF document. I suspect that either:

  • Some features in your document are not within the extents of any map frame.
  • Some of your map frames overlap, and FME is picking the wrong one for features that are in the intersection area.

You may also find it helpful to inspect both your data and the "pdf_frame_metadata" feature types at the same time (in page-points mode), since this makes FME's decision process about coordinate systems a lot clearer.

Did you mean something like this?

 

capture44.png
Did you mean something like this?

 

capture44.png
That is helpful indeed!

 

 

It looks like you have only one map frame, and it doesn't quite include all of the features (something is off to the bottom left, so that one won't have a coordinate system). All of the other features *should* be correctly placed in UTF27-11.

 

 

I'm wondering now whether the coordinate system in the PDF is just poorly defined, and it is either losing some precision or has simply been misconfigured.

 

 

In any case, now that you've attached the data, I can have a look.

 

 

@redgeographics Here is a link with a sample that I will keep valid until this post is solved. 

https://drive.google.com/file/d/1YpDyCUZqvSgxxdrDvSacEsN4wfPw87ex/view?usp=sharing

I have been viewing it in Data Inspector but not with any background map, just the layers in the PDF which include both raster and vector data. I have also run the workspace and viewed the data within ArcMap where I have noticed the offset of 77m about east-southeast from what I presume the actual positions to be. 

/VP  <</Type /Viewport /BBox p0 0 612 792] /Measure 22 0 R>>]
21 0 obj
<<
/Type /PROJCS
/WKT (PROJCSu"unnamed",GEOGCS&"NAD27",DATUM "D_North_America\
n_1927",SPHEROIDS"Clarke_1866",6378206.4,294.9786982139,AUTH\
ORITYo"EPSG","7008"]],AUTHORITY,"EPSG","6267"]],PRIMEM,"Gree\
nwich",0,AUTHORITY<"EPSG","8901"]],DATUM_TRANSFORMO2,2],UNIT\
7"degree",0.0174532925199433],AUTHORITYI"EPSG","4267"]],PROJ\
ECTIONM"Transverse_Mercator"],PARAMETER4"false_easting",5000\
00],PARAMETERP"false_northing",0],PARAMETERM"scale_factor",0\
.9996],PARAMETER0"Central_Meridian",-117],PARAMETERn"Latitud\
e_Of_Origin",0],UNITr"Meter",1]])
>>
endobj
22 0 obj
<<
/Type /Measure
/Subtype /GEO
/Bounds r0 0 0 1 1 1 1 0]
/GPTS  31.28579 -116.02235 31.2805 -115.47811 30.67563 -115.48769 30.6808 
-116.0285]
/LPTS >0 1 1 1 1 0 0 0]
/GCS 21 0 R
>>
endobj 
This is the coordinate system defined in your PDF by the way. It may be helpful for us to compare the WKT with the parameters in the coordinate system detected by FME; there may be a subtle mismatch that is leading to the 77m error.

 

@redgeographics Here is a link with a sample that I will keep valid until this post is solved.

https://drive.google.com/file/d/1YpDyCUZqvSgxxdrDvSacEsN4wfPw87ex/view?usp=sharing

I have been viewing it in Data Inspector but not with any background map, just the layers in the PDF which include both raster and vector data. I have also run the workspace and viewed the data within ArcMap where I have noticed the offset of 77m about east-southeast from what I presume the actual positions to be.

I tried reading in Data Inspector, and it complained when I didn't have a Canada/USA setting for the NAD27->NAD83 datum shift; do you see anything like that?

 

I ran `fme APPLY_SETTINGS SYSTEM "CoordSys/NAD2783 Datum Shift" "usa only"` to fix that (it can also be "canada only", but I suspect "usa only" is correct for your data).

 

 

Also, I understand now why you're seeing messages about LLWGS84->UTM27-11 reprojection: the funny thing about this style of PDF coordinate systems is that the ground control points are in lat-longs no matter what, even if you have some completely different coordinate system like UTM. Anyway, that's why we have to reproject the ground control points to the actual coordinate system.

 

 

I tried reading in Data Inspector, and it complained when I didn't have a Canada/USA setting for the NAD27->NAD83 datum shift; do you see anything like that?

 

I ran `fme APPLY_SETTINGS SYSTEM "CoordSys/NAD2783 Datum Shift" "usa only"` to fix that (it can also be "canada only", but I suspect "usa only" is correct for your data).

 

 

Also, I understand now why you're seeing messages about LLWGS84->UTM27-11 reprojection: the funny thing about this style of PDF coordinate systems is that the ground control points are in lat-longs no matter what, even if you have some completely different coordinate system like UTM. Anyway, that's why we have to reproject the ground control points to the actual coordinate system.

 

 

I did see that complaint (not having the Canada/USA setting for the NAD27->NAD83 datum shift0, and had simply forgotten to mention it. I also assumed that US would be better for my data (located in Baja California).

 

 

I did see that complaint (not having the Canada/USA setting for the NAD27->NAD83 datum shift0, and had simply forgotten to mention it. I also assumed that US would be better for my data (located in Baja California).

 

 

capture45.png

 

capture46.png

 

 

This shows that when I open the shapefile of the same features that I am comparing the PDF data with, the exact same objects have slightly different coordinates as read by FME inspector.

 

 

I tried reading in Data Inspector, and it complained when I didn't have a Canada/USA setting for the NAD27->NAD83 datum shift; do you see anything like that?

 

I ran `fme APPLY_SETTINGS SYSTEM "CoordSys/NAD2783 Datum Shift" "usa only"` to fix that (it can also be "canada only", but I suspect "usa only" is correct for your data).

 

 

Also, I understand now why you're seeing messages about LLWGS84->UTM27-11 reprojection: the funny thing about this style of PDF coordinate systems is that the ground control points are in lat-longs no matter what, even if you have some completely different coordinate system like UTM. Anyway, that's why we have to reproject the ground control points to the actual coordinate system.

 

 

I verified that FME was misplacing my coordinates manually and it is. When I compare the coordinates of the original data set to the shapefile in ArcMAP, the coordinates fall on the shapefiles, not on the FME output.

 

 

capture45.png

 

capture46.png

 

 

This shows that when I open the shapefile of the same features that I am comparing the PDF data with, the exact same objects have slightly different coordinates as read by FME inspector.

 

 

I see! This leads me to few possible scenarios:

 

 

  1. Since PDF only has 32-bit float precision, it is lossy for your data between the conversion from Shapefile to PDF and back. I think this is not too too likely, since you seem to have a consistent error.
  2. The affine matrix we are constructing from your ground control points (`/GPTS` in the comment above) is wrong. This may be due to there being 4 control points that do not form a perfect rectangle. Can you confirm whether there is more/less distortion in different corners/quadrants of the PDF?

 

I tried reading in Data Inspector, and it complained when I didn't have a Canada/USA setting for the NAD27->NAD83 datum shift; do you see anything like that?

 

I ran `fme APPLY_SETTINGS SYSTEM "CoordSys/NAD2783 Datum Shift" "usa only"` to fix that (it can also be "canada only", but I suspect "usa only" is correct for your data).

 

 

Also, I understand now why you're seeing messages about LLWGS84->UTM27-11 reprojection: the funny thing about this style of PDF coordinate systems is that the ground control points are in lat-longs no matter what, even if you have some completely different coordinate system like UTM. Anyway, that's why we have to reproject the ground control points to the actual coordinate system.

 

 

There does seem to be a gradient from 77.2 m off at the most northwestern data point to 75.5 m off at the southeast most point. These are not literally at the corners of the data frame but probably span most of it.

 

 

There does seem to be a gradient from 77.2 m off at the most northwestern data point to 75.5 m off at the southeast most point. These are not literally at the corners of the data frame but probably span most of it.

 

 

Hm, we still seem to have an initial constant offset though. I wonder what the size of a page point is in ground coordinates... just checked, and it's suspiciously similar to our offset (https://www.google.com/search?q=%28644948.6512192317+-+593131.811932617%29+%2F+612&ie;=utf-8&oe;=utf-8&client;=firefox-b-ab)

 

 

Hm, we still seem to have an initial constant offset though. I wonder what the size of a page point is in ground coordinates... just checked, and it's suspiciously similar to our offset (https://www.google.com/search?q=%28644948.6512192317+-+593131.811932617%29+%2F+612&ie;=utf-8&oe;=utf-8&client;=firefox-b-ab)

 

 

I had to disappear for a week to deal with other projects. What does that imply? I am starting to think that I will just be able to use the offset data if I can take the offset into account while using it. However, I am curious if there is a solution to this.

 

 

I had to disappear for a week to deal with other projects. What does that imply? I am starting to think that I will just be able to use the offset data if I can take the offset into account while using it. However, I am curious if there is a solution to this.

 

 

To me it seems to imply there may be an off-by-one error in our code somewhere, or else in the PDF-authoring code that made your data.

 

 

I don't think I can figure out much more unless I have access to the original data and the PDF together, and hopefully some information about the program that created the PDF (although some of that is usually embedded in the PDF).

 

 

Would you mind raising a support ticket with Safe?

 

 

Reply


Community Stats

30,108
Posts
114,294
Replies
37,743
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.