Skip to main content

Hi,

 

I have a dataset in .dbf format, with 885 rows. When I read the file in FME it reads the 885 rows, but when I use the DuplicateFilter it tells me I have 330 unique rows and the rest are duplicates.

 

But actually all these 885 rows are unique, I can see that when I look at the file using ArcGIS Pro, and my colleague have looked at the file in another text-application. So it seems FME keeps the total number correct, 885, but changes some unique numbers so they are duplicates somehow.

 

In AGP:

imageIn FME:

image 

How can this happen? I have tried reading the file both using the "normal" .dbf-writer, as well zipping the file and using Creator > ParameterFetcher > FeatureReader > FeatureReader > AttributeExposer.

 

 

Hard to say without data :)

How is the DuplicateFilter configured?


Hi @nielsgerrits​ 

I have added data here.


Hi @nielsgerrits​ 

I have added data here.

I could reproduce your issue in FME. When I open the file using Excel or QGIS I see the same results, but can't open it in ArcGIS Pro.

 

Not sure what is wrong here. Is it a possibility that the file is open in an edit session in ArcGIS Pro, which has a lock on it? Or not saved edits?

 

It feels like the issue is not in FME / Excel / QGIS, but rather something else is wrong. I have seen people edit the dbf of a shapefile set using Excel causing weird issues like this, not sure if this causes your issue.

 

You always can try to export the data from ArcGIS Pro to a new file to see if the problem goes away.


I could reproduce your issue in FME. When I open the file using Excel or QGIS I see the same results, but can't open it in ArcGIS Pro.

 

Not sure what is wrong here. Is it a possibility that the file is open in an edit session in ArcGIS Pro, which has a lock on it? Or not saved edits?

 

It feels like the issue is not in FME / Excel / QGIS, but rather something else is wrong. I have seen people edit the dbf of a shapefile set using Excel causing weird issues like this, not sure if this causes your issue.

 

You always can try to export the data from ArcGIS Pro to a new file to see if the problem goes away.

Hi @nielsgerrits​ ,

good that you were able to reproduce.

I didn't have the dbf file open in ArcGIS before I needed to check it, so I don't think it can be that. I could of course try to export it from there, but I am building a FME- script that will be published to the server, and it will read dbf-files directly. The script is developed so that other people can do the update (without any pre-processing). So really need to figure this out.


Hi @Anne Kathrine Wenaas Ribe​ 

Can you open the file in Text Editor such as Notepad++?

 

 

 


Hi @Anne Kathrine Wenaas Ribe​ 

Can you open the file in Text Editor such as Notepad++?

 

 

 

Hi @rahulsharma​ ,

The file I uploaded was a "stripped" one without sensitive data. When I open the original file I can see the unique values.

 

@nielsgerrits​ : I was also able to open it in excel, was it the duplicates you saw there or the unique values? I see the unique values.

 

Since I can't share the original data Im not sure if you need to test anything more. I might have to proceed to figure out a solution for this.


Hi @rahulsharma​ ,

The file I uploaded was a "stripped" one without sensitive data. When I open the original file I can see the unique values.

 

@nielsgerrits​ : I was also able to open it in excel, was it the duplicates you saw there or the unique values? I see the unique values.

 

Since I can't share the original data Im not sure if you need to test anything more. I might have to proceed to figure out a solution for this.

If I open the uploaded .dbf in Excel or Notepad++ I see duplicate values, so I suspect your sample file is different from the file in the printscreen in ArcGIS Pro.


Hi @rahulsharma​ ,

The file I uploaded was a "stripped" one without sensitive data. When I open the original file I can see the unique values.

 

@nielsgerrits​ : I was also able to open it in excel, was it the duplicates you saw there or the unique values? I see the unique values.

 

Since I can't share the original data Im not sure if you need to test anything more. I might have to proceed to figure out a solution for this.

Yes, I get a similar result at @nielsgerrits​ i.e. duplicate values when I open the data in Notepad++

Can I request you @Anne Kathrine Wenaas Ribe​ to open the attached data (not the original you have as I can not vouch for that dataset) in the notepad editor?

If you think it is happening to your dataset (a number of tables) we can follow up via a support ticket instead?

 


Hi @rahulsharma​ ,

The file I uploaded was a "stripped" one without sensitive data. When I open the original file I can see the unique values.

 

@nielsgerrits​ : I was also able to open it in excel, was it the duplicates you saw there or the unique values? I see the unique values.

 

Since I can't share the original data Im not sure if you need to test anything more. I might have to proceed to figure out a solution for this.

Hi @rahulsharma​ ,

 

I opened my original file in notepad++ and it looks correct there with all the unique values. So it seems multiple softwares reads the data correct, besides FME.

 

I asked my project manager if it was okey to share my original data with you, as the copy I made is not a good representative. That was ok, so Ill make a support case from it.


I had a different problem with certain shapefile dbfs, where a random amount of features had a value of 0 in FME instead of their actual values as seen in QGIS and Arcmap. Reported as C685504 - Shapefile attributes not returning correct value

This bug was present in all versions of FME 2020-2022, but not present in FME 2019 (build 19813), and also fixed in FME beta 2023 (build 23210).

Can you try the workflow in the 2023 beta?


Reply