Skip to main content
Solved

Does FME support Inserts into Hive tables using JDBC


Best answer by ravenkopelman

@TiaAtSafe is correct that FME does not include a Hive reader or writer. However, this is likely achievable through our JDBC format.

In 2015 we performed reader experiments with three Hive JDBC drivers.

Progress/DataDirect: Did not work due to auto-commit conflicts, but between FME improvements since then and the later-discovered TransactionMode driver flag I believe it is viable.

Cloudera: Worked as long as the driver classname was explicitly specified.

Apache: Unable to get working, but this project has advanced since then. There may be hope.

We did not perform writer experiments because Hive at that time did not support standard SQL INSERT statements, so you would have needed to manually send your insert statements through a SqlExecutor transformer. It appears that lack of standard INSERT support may also have improved over the last few years.

In short, if you are willing to put the effort in you can probably get this working with the JDBC tools currently available.

View original
Did this help you find an answer to your question?

3 replies

fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • July 18, 2017

Hi @rfslivajr, FME currently does not support writing to Hive (please check out the table in Using the FME JDBC). I recommend voting on the Hadoop idea on our Ideas page to increase the awareness of the suggestion.


ravenkopelman
Safer
Forum|alt.badge.img+1
  • Safer
  • Best Answer
  • July 26, 2017

@TiaAtSafe is correct that FME does not include a Hive reader or writer. However, this is likely achievable through our JDBC format.

In 2015 we performed reader experiments with three Hive JDBC drivers.

Progress/DataDirect: Did not work due to auto-commit conflicts, but between FME improvements since then and the later-discovered TransactionMode driver flag I believe it is viable.

Cloudera: Worked as long as the driver classname was explicitly specified.

Apache: Unable to get working, but this project has advanced since then. There may be hope.

We did not perform writer experiments because Hive at that time did not support standard SQL INSERT statements, so you would have needed to manually send your insert statements through a SqlExecutor transformer. It appears that lack of standard INSERT support may also have improved over the last few years.

In short, if you are willing to put the effort in you can probably get this working with the JDBC tools currently available.


ravenkopelman
Safer
Forum|alt.badge.img+1
ravenkopelman wrote:

@TiaAtSafe is correct that FME does not include a Hive reader or writer. However, this is likely achievable through our JDBC format.

In 2015 we performed reader experiments with three Hive JDBC drivers.

Progress/DataDirect: Did not work due to auto-commit conflicts, but between FME improvements since then and the later-discovered TransactionMode driver flag I believe it is viable.

Cloudera: Worked as long as the driver classname was explicitly specified.

Apache: Unable to get working, but this project has advanced since then. There may be hope.

We did not perform writer experiments because Hive at that time did not support standard SQL INSERT statements, so you would have needed to manually send your insert statements through a SqlExecutor transformer. It appears that lack of standard INSERT support may also have improved over the last few years.

In short, if you are willing to put the effort in you can probably get this working with the JDBC tools currently available.

@rfslivajr FME 2018.1 will contain a Hive Reader that will also make it easier to send INSERT statements through the SqlExecutor. We have a Writer tentatively scheduled for FME 2019.0; please raise an issue and reference PR#82991 if you would like status updates.

 

 

 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings