Solved

Reading AGS/Text file in FME

R

Hello,

 

I am trying to create a reader to process AGS 4 data but am struggling with it a bit.

For those who haven't encountered the data before, I have included an example below.

When opened as a text file, the data is all in one column. It is grouped into sections that define a subsection of the overall dataset. These sections always start with the "GROUP" header and end with a empty line before the next group header.

Fortunately, the groups always follow the same pattern:

"GROUP" = Name of the group

"HEADING" = The different data headers (These would be the column headers if brought into excel)

"UNIT"= The Unit of the header

"TYPE" = The Data type

"DATA" = The actual data for each header.

 

"GROUP","GCHM"

"HEADING","LOCA_ID","SAMP_TOP","SAMP_REF","SAMP_TYPE","SAMP_ID","SPEC_REF","SPEC_DPTH","GCHM_CODE","GCHM_METH","GCHM_TTYP","GCHM_RESL","GCHM_UNIT","GCHM_NAME","SPEC_DESC","SPEC_PREP","GCHM_REM","GCHM_LAB","GCHM_CRED","TEST_STAT","FILE_FSET"

"UNIT","","m","","","","","m","","","","","","","","","","","","",""

"TYPE","ID","2DP","X","PA","ID","X","2DP","PA","X","PA","X","PU","X","X","X","X","X","X","X","X"

"DATA","BH502","3.00","11","W","","","","PH","","WATER","7.2","-","pH","Stiff grey slightly sandy CLAY with some fine to coarse subrounded gravel","","","CHEMLAB","","Final",""

"DATA","BH502","3.00","11","W","","","","WS","BS1377-3","WATER","0.037","g/l","Water soluble sulphate","Stiff grey slightly sandy CLAY with some fine to coarse subrounded gravel","","","CHEMLAB","","Final",""

"DATA","BH502","3.00","11","W","","","","CL","BS1377-3","WATER","51","mg/l","Water soluble chloride","Stiff grey slightly sandy CLAY with some fine to coarse subrounded gravel","","","CHEMLAB","","Final",""

"DATA","BH502","3.00","10","W","","","","WS","BS1377-3","WATER","0.04","g/l","Water soluble sulphate","Stiff grey slightly sandy CLAY with some fine to coarse subrounded gravel","","","CHEMLAB","","Final",""

 

Thanks in advance, this is a tough one!

 

icon

Best answer by daveatsafe 8 May 2020, 00:42

View original

13 replies

ebygomm
Rank
Userlevel 1
Badge +10

What are you looking for as an end result? Something like this?

daveatsafe
Rank
Userlevel 2
Badge +17

Hi @rjwillshear,

I am attaching a workspace that will read the text file and output to an individual CSV file per group, using the headings and data types specified in the header lines for each group.

The workspace creates a schema feature for use with Dynamic writing, so you can replace the CSV writer with any writer that supports no-geometry features. Just enable Dynamic Schema Definition and set Schema source to 'Schema from Schema Feature'.

Unfortunately, the reliance on dynamic schema prevents me from converting the workspace to a custom reader, since those presently require fixed schemas.

Please try the workspace and let me know if it reads your files correctly.

 

ags4reader.fmw
daveatsafe
Rank
Userlevel 2
Badge +17

Hi @rjwillshear,

Tried to write to Excel instead of CSV on a downloaded test .ags file from http://www.agsdataformat.com/datatransferv4/download.php, which illustrated a few problems in the earlier workspace. Please try this one.

AGS4Reader.fmw

R

Hi @rjwillshear,

Tried to write to Excel instead of CSV on a downloaded test .ags file from http://www.agsdataformat.com/datatransferv4/download.php, which illustrated a few problems in the earlier workspace. Please try this one.

AGS4Reader.fmw

Yes, that absolutely works! Amazing thank you, having it in that format makes it much easier to convert to GIS or database features

R

What are you looking for as an end result? Something like this?

Very close yes, thank you. That format would be good for adding to a GIS layer.

R

Yes, that absolutely works! Amazing thank you, having it in that format makes it much easier to convert to GIS or database features

Hi @daveatsafe,

As a follow up, just tried the workflow on some live data at work. It works perfectly, doesn't seem to be tripped up by some of the more obscure data or custom fields we have. I can't thank you enough.

G

Hi @rjwillshear,

Tried to write to Excel instead of CSV on a downloaded test .ags file from http://www.agsdataformat.com/datatransferv4/download.php, which illustrated a few problems in the earlier workspace. Please try this one.

AGS4Reader.fmw

Hi @daveatsafe / @rjwillshear

Could you confirm what steps were done before the FME output ran? I have tried to run it with the file in excel format but does not seem to work for me.

Appreciate your help

R

Hi @rjwillshear,

Tried to write to Excel instead of CSV on a downloaded test .ags file from http://www.agsdataformat.com/datatransferv4/download.php, which illustrated a few problems in the earlier workspace. Please try this one.

AGS4Reader.fmw

Hi @daveatsafe​,

 

Is there anyway to reupload the file you provided, it has been deleted from our system.

 

Thanks

 

Ryan

R

Hi @daveatsafe / @rjwillshear

Could you confirm what steps were done before the FME output ran? I have tried to run it with the file in excel format but does not seem to work for me.

Appreciate your help

This is designed to run on the raw AGS 4 file, not an excel version.

M
Badge +2

AGS4 reader is also available from @daveatsafe on the FME HUB

daveatsafe
Rank
Userlevel 2
Badge +17

Hi @daveatsafe​,

 

Is there anyway to reupload the file you provided, it has been deleted from our system.

 

Thanks

 

Ryan

Hi @rjwillshear​ ,

We have fixed the link to the workspace I posted above. You should be able to download it now.

R

Hi @daveatsafe​,

 

Is there anyway to reupload the file you provided, it has been deleted from our system.

 

Thanks

 

Ryan

Much appreciated!

johnk
Rank

Hi @daveatsafe​,

 

Is there anyway to reupload the file you provided, it has been deleted from our system.

 

Thanks

 

Ryan

Hi, please can I ask what the output from this workspace is supposed to look like? I've tried to run an AGS file through this and I get a very strange output

Reply


Community Stats

28,670
Posts
109,364
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.