I'm calling a child Workspace using a WorkspaceRunner. The child Workspace says translation was successful as does the WorkspaceRunner, but when I reviewed the log for the child Workspace, I noticed there were warnings like this: "Provider error `(-2147217873) Violation of PRIMARY KEY constraint 'xxxx'. Cannot insert duplicate key in object 'yyyy'". If this happens, I want the Workspace to fail/terminate. Is there a way I can do this? Would a FeatureWriter transformer rather than a normal Writer give me this functionality?
Which writer format is this? Pure SQL Server or is there some ArcSDE in the mix as well?
Which writer format is this? Pure SQL Server or is there some ArcSDE in the mix as well?
Â
There's a little known trick to catching almost any message in the FME log and to act on it, using the fmeobjects.FMELogFile callback method.
Try putting the following in the Python startup script:
import fmeobjects
global AN_ERROR_OCCURRED
AN_ERROR_OCCURREDÂ =Â None
def LogSkimmer(severity, text):
    if text.find('Violation of PRIMARY KEY constraint') > -1:
        global AN_ERROR_OCCURRED
        AN_ERROR_OCCURRED = text
fmeobjects.FMELogFile().setCallBack(LogSkimmer)
Then just before your SQL Server Spatial writer, insert a PythonCaller with the following code:
import fmeobjects
def detect_errors(feature):
    global AN_ERROR_OCCURRED
    if AN_ERROR_OCCURRED:
        raise fmeobjects.FMEException(AN_ERROR_OCCURRED)
Let me know how that works for your case.
There's a little known trick to catching almost any message in the FME log and to act on it, using the fmeobjects.FMELogFile callback method.
Try putting the following in the Python startup script:
import fmeobjects
global AN_ERROR_OCCURRED
AN_ERROR_OCCURREDÂ =Â None
def LogSkimmer(severity, text):
    if text.find('Violation of PRIMARY KEY constraint') > -1:
        global AN_ERROR_OCCURRED
        AN_ERROR_OCCURRED = text
fmeobjects.FMELogFile().setCallBack(LogSkimmer)
Then just before your SQL Server Spatial writer, insert a PythonCaller with the following code:
import fmeobjects
def detect_errors(feature):
    global AN_ERROR_OCCURRED
    if AN_ERROR_OCCURRED:
        raise fmeobjects.FMEException(AN_ERROR_OCCURRED)
Let me know how that works for your case.
Â
Â
Â
Â
If you set the transaction to a number, larger than the number of records you are inserting, the transaction will be rolled back by any database failure.
The log file will contain the message, returned by the database.
I am 100% sure this is how it works for Oracle, and more than 99% sure it works the same for SQL Server.
Hope this helps.
Â
Â
If you set the transaction to a number, larger than the number of records you are inserting, the transaction will be rolled back by any database failure.
The log file will contain the message, returned by the database.
I am 100% sure this is how it works for Oracle, and more than 99% sure it works the same for SQL Server.
Hope this helps.
Â
Use a sql executor to get all constraints (usually at begin of your script) Process them and make your data respect the constraints before trying to write it to the table(s).
Here is a oracle sql version
select search_condition,table_name
Â
from user_constraintsÂ
where table_name =Â
and constraint_type ='C'Choose the constraint type you require. You might need to access all_constraints in stead of the user_constraintsUse a sql executor to get all constraints (usually at begin of your script) Process them and make your data respect the constraints before trying to write it to the table(s).
Here is a oracle sql version
select search_condition,table_name
Â
from user_constraintsÂ
where table_name =Â
and constraint_type ='C'Choose the constraint type you require. You might need to access all_constraints in stead of the user_constraintsUse a sql executor to get all constraints (usually at begin of your script) Process them and make your data respect the constraints before trying to write it to the table(s).
Here is a oracle sql version
select search_condition,table_name
Â
from user_constraintsÂ
where table_name =Â
and constraint_type ='C'Choose the constraint type you require. You might need to access all_constraints in stead of the user_constraintsÂ
Â
Your idea has got me thinking that I could put a DuplicateFilter in for the primary key attribute and connect the Duplicate port to a Terminator. The Python code option might make for a quicker translation though....Â
Â
Plus the DuplicateFilter would only pick up features being processed, not conflicts with data already in the SQL table.Â
Â
Â
Your idea has got me thinking that I could put a DuplicateFilter in for the primary key attribute and connect the Duplicate port to a Terminator. The Python code option might make for a quicker translation though....Â
Â
Plus the DuplicateFilter would only pick up features being processed, not conflicts with data already in the SQL table.Â
There's a little known trick to catching almost any message in the FME log and to act on it, using the fmeobjects.FMELogFile callback method.
Try putting the following in the Python startup script:
import fmeobjects
global AN_ERROR_OCCURRED
AN_ERROR_OCCURREDÂ =Â None
def LogSkimmer(severity, text):
    if text.find('Violation of PRIMARY KEY constraint') > -1:
        global AN_ERROR_OCCURRED
        AN_ERROR_OCCURRED = text
fmeobjects.FMELogFile().setCallBack(LogSkimmer)
Then just before your SQL Server Spatial writer, insert a PythonCaller with the following code:
import fmeobjects
def detect_errors(feature):
    global AN_ERROR_OCCURRED
    if AN_ERROR_OCCURRED:
        raise fmeobjects.FMEException(AN_ERROR_OCCURRED)
Let me know how that works for your case.
Â
Â
Also, it says 1 feature was written when it clearly wasn't.Â
Â
2018-09-21 14:39:41|  4.0|  0.1|WARN  |Microsoft SQL Server Spatial Writer: Failed to write a feature of type `yyyy' to the database. Provider error `(-2147217873) Violation of PRIMARY KEY constraint 'xxxx'. Cannot insert duplicate key in object 'yyyy'. The duplicate key value is (1234).'. SQL Command `DECLARE @WKB0 varbinary(max); SET @WKB0 = ?; DECLARE @SRID0 INTEGER; SET @SRID0 = ?; INSERT INTO yyyy (ifield1], Âfield2], etc]) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CASE @WKB0 WHEN '' THEN NULL ELSE geometry::STGeomFromWKB(@WKB0,@SRID0) END)'Â
...Â
2018-09-21 14:39:41|  4.0|  0.0|ERROR |Microsoft SQL Server Spatial Writer: Failed to write a feature of type `yyyy' to the database. Provider error `(-2147217873) Violation of PRIMARY KEY constraint 'xxxx'. Cannot insert duplicate key in object 'yyyy'. The duplicate key value is (1234).'. SQL Command `DECLARE @WKB0 varbinary(max); SET @WKB0 = ?; DECLARE @SRID0 INTEGER; SET @SRID0 = ?; INSERT INTO yyyy (efield1], efield2], eetc]) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CASE @WKB0 WHEN '' THEN NULL ELSE geometry::STGeomFromWKB(@WKB0,@SRID0) END)'Â
2018-09-21 14:39:41|  4.0|  0.0|ERROR |Error encountered while calling function `detect_errors'Â
2018-09-21 14:39:41|  4.0|  0.0|FATAL |PythonCaller(PythonFactory): PythonFactory failed to process featureÂ
2018-09-21 14:39:41|  4.0|  0.0|ERROR |A fatal error has occurred. Check the logfile above for detailsÂ
2018-09-21Â 14:39:41|Â Â 4.0|Â Â 0.0|ERRORÂ |Â
2018-09-21 14:39:41|  4.0|  0.0|INFORM|Microsoft SQL Server Spatial Writer: Closing `zzzz' . Write operation completeÂ
2018-09-21Â 14:39:41|Â Â 4.0|Â Â 0.0|INFORM|=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41|  4.0|  0.0|INFORM|Feature output statistics for `MSSQL_SPATIAL' writer using keyword `MSSQL_SPATIAL_2':Â
2018-09-21Â 14:39:41|Â Â 4.0|Â Â 0.0|STATSÂ |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41|  4.0|  0.0|STATS |  Features WrittenÂ
2018-09-21Â 14:39:41|Â Â 4.0|Â Â 0.0|STATSÂ |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41|  4.0|  0.0|STATS |yyyy  1Â
2018-09-21Â 14:39:41|Â Â 4.0|Â Â 0.0|STATSÂ |==============================================================================Â
2018-09-21 14:39:41|  4.0|  0.0|STATS |Total Features WrittenÂÂ
Â
Â
Also, it says 1 feature was written when it clearly wasn't.Â
Â
2018-09-21 14:39:41| 4.0| 0.1|WARN |Microsoft SQL Server Spatial Writer: Failed to write a feature of type `yyyy' to the database. Provider error `(-2147217873) Violation of PRIMARY KEY constraint 'xxxx'. Cannot insert duplicate key in object 'yyyy'. The duplicate key value is (1234).'. SQL Command `DECLARE @WKB0 varbinary(max); SET @WKB0 = ?; DECLARE @SRID0 INTEGER; SET @SRID0 = ?; INSERT INTO yyyy (yfield1], 1field2], 2etc]) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CASE @WKB0 WHEN '' THEN NULL ELSE geometry::STGeomFromWKB(@WKB0,@SRID0) END)'Â
...Â
2018-09-21 14:39:41| 4.0| 0.0|ERROR |Microsoft SQL Server Spatial Writer: Failed to write a feature of type `yyyy' to the database. Provider error `(-2147217873) Violation of PRIMARY KEY constraint 'xxxx'. Cannot insert duplicate key in object 'yyyy'. The duplicate key value is (1234).'. SQL Command `DECLARE @WKB0 varbinary(max); SET @WKB0 = ?; DECLARE @SRID0 INTEGER; SET @SRID0 = ?; INSERT INTO yyyy (yfield1], lfield2], letc]) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CASE @WKB0 WHEN '' THEN NULL ELSE geometry::STGeomFromWKB(@WKB0,@SRID0) END)'Â
2018-09-21 14:39:41| 4.0| 0.0|ERROR |Error encountered while calling function `detect_errors'Â
2018-09-21 14:39:41| 4.0| 0.0|FATAL |PythonCaller(PythonFactory): PythonFactory failed to process featureÂ
2018-09-21 14:39:41| 4.0| 0.0|ERROR |A fatal error has occurred. Check the logfile above for detailsÂ
2018-09-21 14:39:41| 4.0| 0.0|ERROR |Â
2018-09-21 14:39:41| 4.0| 0.0|INFORM|Microsoft SQL Server Spatial Writer: Closing `zzzz' . Write operation completeÂ
2018-09-21 14:39:41| 4.0| 0.0|INFORM|=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41| 4.0| 0.0|INFORM|Feature output statistics for `MSSQL_SPATIAL' writer using keyword `MSSQL_SPATIAL_2':Â
2018-09-21 14:39:41| 4.0| 0.0|STATS |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41| 4.0| 0.0|STATS | Features WrittenÂ
2018-09-21 14:39:41| 4.0| 0.0|STATS |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41| 4.0| 0.0|STATS |yyyy 1Â
2018-09-21 14:39:41| 4.0| 0.0|STATS |==============================================================================Â
2018-09-21 14:39:41| 4.0| 0.0|STATS |Total Features WrittenÂ
Â
Since we're basically overriding the default FME termination behavior, I'm not too surprised that the "total features written" isn't quite correct.Â
Â
Also, it says 1 feature was written when it clearly wasn't.Â
Â
2018-09-21 14:39:41| 4.0| 0.1|WARN |Microsoft SQL Server Spatial Writer: Failed to write a feature of type `yyyy' to the database. Provider error `(-2147217873) Violation of PRIMARY KEY constraint 'xxxx'. Cannot insert duplicate key in object 'yyyy'. The duplicate key value is (1234).'. SQL Command `DECLARE @WKB0 varbinary(max); SET @WKB0 = ?; DECLARE @SRID0 INTEGER; SET @SRID0 = ?; INSERT INTO yyyy (yfield1], 1field2], 2etc]) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CASE @WKB0 WHEN '' THEN NULL ELSE geometry::STGeomFromWKB(@WKB0,@SRID0) END)'Â
...Â
2018-09-21 14:39:41| 4.0| 0.0|ERROR |Microsoft SQL Server Spatial Writer: Failed to write a feature of type `yyyy' to the database. Provider error `(-2147217873) Violation of PRIMARY KEY constraint 'xxxx'. Cannot insert duplicate key in object 'yyyy'. The duplicate key value is (1234).'. SQL Command `DECLARE @WKB0 varbinary(max); SET @WKB0 = ?; DECLARE @SRID0 INTEGER; SET @SRID0 = ?; INSERT INTO yyyy (yfield1], lfield2], letc]) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, CASE @WKB0 WHEN '' THEN NULL ELSE geometry::STGeomFromWKB(@WKB0,@SRID0) END)'Â
2018-09-21 14:39:41| 4.0| 0.0|ERROR |Error encountered while calling function `detect_errors'Â
2018-09-21 14:39:41| 4.0| 0.0|FATAL |PythonCaller(PythonFactory): PythonFactory failed to process featureÂ
2018-09-21 14:39:41| 4.0| 0.0|ERROR |A fatal error has occurred. Check the logfile above for detailsÂ
2018-09-21 14:39:41| 4.0| 0.0|ERROR |Â
2018-09-21 14:39:41| 4.0| 0.0|INFORM|Microsoft SQL Server Spatial Writer: Closing `zzzz' . Write operation completeÂ
2018-09-21 14:39:41| 4.0| 0.0|INFORM|=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41| 4.0| 0.0|INFORM|Feature output statistics for `MSSQL_SPATIAL' writer using keyword `MSSQL_SPATIAL_2':Â
2018-09-21 14:39:41| 4.0| 0.0|STATS |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41| 4.0| 0.0|STATS | Features WrittenÂ
2018-09-21 14:39:41| 4.0| 0.0|STATS |=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-Â
2018-09-21 14:39:41| 4.0| 0.0|STATS |yyyy 1Â
2018-09-21 14:39:41| 4.0| 0.0|STATS |==============================================================================Â
2018-09-21 14:39:41| 4.0| 0.0|STATS |Total Features WrittenÂ
Â
Since we're basically overriding the default FME termination behavior, I'm not too surprised that the "total features written" isn't quite correct.Â
A fatal error has occurred. Check the logfile above for detailsProgram Terminating
Â
Translation FAILED.
Â
Â
Regarding features written, it's not the overriding of FME behaviour that's the issue. If I run the translation without the Python, FME still says it's written the features, even though they all failed to write to the SQL Server table because of a Primary Key violation. I know we're talking semantics about what "written" means here, but I would argue that if it's not written, it's not written! I'm less decided about whether some features violating a Primary Key constraint but not others should constitute a successful or failed translation...Â
Â
A fatal error has occurred. Check the logfile above for detailsProgram Terminating
Â
Translation FAILED.
Â
Â
Regarding features written, it's not the overriding of FME behaviour that's the issue. If I run the translation without the Python, FME still says it's written the features, even though they all failed to write to the SQL Server table because of a Primary Key violation. I know we're talking semantics about what "written" means here, but I would argue that if it's not written, it's not written! I'm less decided about whether some features violating a Primary Key constraint but not others should constitute a successful or failed translation...Â
Â
Out of curiosity, what happens if you use a FeatureWriter? Does the offending feature exit the <Rejected> port?Â
A fatal error has occurred. Check the logfile above for detailsProgram Terminating
Â
Translation FAILED.
Â
Â
Regarding features written, it's not the overriding of FME behaviour that's the issue. If I run the translation without the Python, FME still says it's written the features, even though they all failed to write to the SQL Server table because of a Primary Key violation. I know we're talking semantics about what "written" means here, but I would argue that if it's not written, it's not written! I'm less decided about whether some features violating a Primary Key constraint but not others should constitute a successful or failed translation...Â
Â
Â
Â
A fatal error has occurred. Check the logfile above for detailsProgram Terminating
Â
Translation FAILED.
Â
Â
Regarding features written, it's not the overriding of FME behaviour that's the issue. If I run the translation without the Python, FME still says it's written the features, even though they all failed to write to the SQL Server table because of a Primary Key violation. I know we're talking semantics about what "written" means here, but I would argue that if it's not written, it's not written! I'm less decided about whether some features violating a Primary Key constraint but not others should constitute a successful or failed translation...Â
Â
There's a little known trick to catching almost any message in the FME log and to act on it, using the fmeobjects.FMELogFile callback method.
Try putting the following in the Python startup script:
import fmeobjects
global AN_ERROR_OCCURRED
AN_ERROR_OCCURREDÂ =Â None
def LogSkimmer(severity, text):
    if text.find('Violation of PRIMARY KEY constraint') > -1:
        global AN_ERROR_OCCURRED
        AN_ERROR_OCCURRED = text
fmeobjects.FMELogFile().setCallBack(LogSkimmer)
Then just before your SQL Server Spatial writer, insert a PythonCaller with the following code:
import fmeobjects
def detect_errors(feature):
    global AN_ERROR_OCCURRED
    if AN_ERROR_OCCURRED:
        raise fmeobjects.FMEException(AN_ERROR_OCCURRED)
Let me know how that works for your case.
Â
2018-10-08 11:58:17|  18.8|  0.0|INFORM|Optional `close' method not present; not calledÂ
I've looked this up and read a couple of forum topics about it but I'm not sure I completely understand what's going on.  I assume it has something to do with the fact that a Writer is the end of a translation whereas the translation can continue after a FeatureWriter.  It doesn't look as though anything bad is happening, but is there something I need to add to the Python script as best practice?Â
Â
BTW, this is for a successful translation.  I haven't yet tested whether Primary Key violations still terminate the translation.Â
Â
2018-10-08 11:58:17|  18.8|  0.0|INFORM|Optional `close' method not present; not calledÂ
I've looked this up and read a couple of forum topics about it but I'm not sure I completely understand what's going on.  I assume it has something to do with the fact that a Writer is the end of a translation whereas the translation can continue after a FeatureWriter.  It doesn't look as though anything bad is happening, but is there something I need to add to the Python script as best practice?Â
Â
BTW, this is for a successful translation.  I haven't yet tested whether Primary Key violations still terminate the translation.Â
Â
The log message seems to indicate that you have a class method that's missing the close() method. This is nothing to worry about, it's just for information. You can also get rid of the messge by adding the following method to your class definition:Â
    def close(self):
        pass
Â
The log message seems to indicate that you have a class method that's missing the close() method. This is nothing to worry about, it's just for information. You can also get rid of the messge by adding the following method to your class definition:Â
    def close(self):
        pass
Â
Â
Â
Â
Regarding the line in the job log about the optional 'close' method not being present, you can safely ignore that.