jdbc_fetch_size no effect?

[LxiaoGirl]

logstash 1.5.5
jdbc 1.0.0

error log:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid24312.hprof ...
Heap dump file created [1159466364 bytes in 21.477 secs]

i read
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-jdbc_page_size
add parameter jdbc_fetch_size.
but,no effect.
I want to increase memory?

[jaykay153]

This isn't working for me either using postgresql. Looking at the postgres documentation, the code needs to explicitly set the connection autocommit mode to false e.g. conn.setAutoCommit(false);
https://jdbc.postgresql.org/documentation/91/query.html#fetchsize-example

For data size of 10 million plus, this is a must (jdbc_paging_enabled is too slow).

[jaykay153]

Can this be prioritized please?

[suyograo]

thanks for the issue, we'll get this soon.

[LxiaoGirl]

fixed?

[edeak]

I guess pagination could be made faster if the following statement is executed:
select t.* from (select id from t where id > :LAST_ID limit :LIMIT) q join t on q.id = t.id

I use this trick to iterate through millions of records and its quite fast, the only thing to be solved is getting the last id and use it as a variable inside the plugin.

[untergeek]

the only thing to be solved is getting the last id and use it as a variable inside the plugin.

This was committed in #108 and can be gained by updating to version 3.0.0 of the plugin.

Some interim documentation: #57 (comment)

[jaykay153]

Many tables don't have the last ID column. What about the jdbc_fetch_size fix please?

[jjunqueira]

+1, please fix!

The paging is slower and because of the way that the count query is implemented for paging(just by wrapping the defined statement) I am seeing Incorrect Key File errors in MySQL.

An alternative might also be allowing a way to define the count query statement?

Let me know if I can do anything to help. Thank you for your hard work on this project!

[rspuler]

Serious problem for me too. This plugin does not work with postgresql driver and full import of big tables. Allways results in an OutOfMemoryError due to the reason that jaykay153 has given.

[rspuler]

Workaround is not to use the official postgresql drivers but this one here:
https://github.com/impossibl/pgjdbc-ng which seams to work (probably be setting autocommit to false, when fetchSize is set)

[axhiao]

It seems that an additional parameter ?useCursorFetch=true needs to be added to the connection string of mysql 5.x.