Skip to content

Commit 9703e86

Browse files
jcustenborderjcustenborder
authored
Merge pull request #2 from jcustenborder/jenkins
Splunk Source Connector
2 parents 558b2e0 + 74183a0 commit 9703e86

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+1360
-193
lines changed

‎README.md‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,37 @@ This connector support receiving data and writing data to Splunk.
33

44
# Source Connector
55

6-
This is under development.
6+
The Splunk Source connector allows emulates a [Splunk Http Event Collector](http://dev.splunk.com/view/event-collector/SP-CAAAE6M) to allow
7+
application that normally log to Splunk to instead write to Kafka. The goal of this plugin is to make the change nearly
8+
transparent to the user. This plugin currently has support for [X-Forwarded-For](https://en.wikipedia.org/wiki/X-Forwarded-For) so
9+
it will sit behind a load balancer nicely.
10+
11+
## Configuration
12+
13+
| Name | Description | Type | Default | Valid Values | Importance |
14+
|----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|--------------|------------|
15+
| kafka.topic | This value contains the topic that the messages will be written to. If topic per index is enabled this will be the prefix for the topic. If not this will be the exact topic. | string | | | high |
16+
| splunk.collector.index.default | The index that will be used if no index is specified in the event message. | string | | | high |
17+
| splunk.ssl.key.store.password | The password for opening the keystore. | password | | | high |
18+
| splunk.ssl.key.store.path | The path to the keystore on the local filesystem. | string | | | high |
19+
| splunk.port | The port to configure the http listener on. | int | 8088 | | high |
20+
| topic.per.index | Flag determines if the all generated messages should be written toa single topic or should the messages be placed in a topic prefixed by the supplied index. If true the `kafka.topic` setting will be concatenated along with the index name. If false the `kafka.topic` value will be used for the topic. | boolean | false | | medium |
21+
| backoff.ms | The number of milliseconds to back off when there are no records in thequeue. | int | 100 | | low |
22+
| batch.size | Maximum number of records to write per poll call. | int | 10000 | | low |
23+
| splunk.collector.index.allowed | The indexes this connector allows data to be written for. Specifying an index outside of this list will result in an exception being raised. | list | [] | | low |
24+
| splunk.collector.url | Path fragement the servlet should respond on | string | /services/collector/event | | low |
25+
| splunk.ssl.renegotiation.allowed | Flag to determine if ssl renegotiation is allowed. | boolean | true | | low |
26+
27+
### Example Config
28+
29+
```
30+
name=splunk-http-source
31+
tasks.max=1
32+
connector.class=io.confluent.kafka.connect.splunk.SplunkHttpSourceConnector
33+
splunk.ssl.key.store.path=/etc/security/keystore.jks
34+
splunk.ssl.key.store.password=password
35+
splunk.collector.index.default=main
36+
```
737

838
# Sink Connector
939

@@ -52,7 +82,7 @@ Here is an example of an event generated by [Kafka Connect Syslog](https://githu
5282

5383
```
5484
{
55-
"host": "vpn.custenborder.com",
85+
"host": "vpn.example.com",
5686
"time": 1472342182,
5787
"event": {
5888
"charset": "UTF-8",

‎bin/debug.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
#!/usr/bin/env bash
22
#
3-
# Copyright (C) ${project.inceptionYear} Jeremy Custenborder ([email protected])
3+
# Copyright © 2016 Jeremy Custenborder ([email protected])
44
#
55
# Licensed under the Apache License, Version 2.0 (the "License");
66
# you may not use this file except in compliance with the License.
77
# You may obtain a copy of the License at
88
#
9-
# http://www.apache.org/licenses/LICENSE-2.0
9+
# http://www.apache.org/licenses/LICENSE-2.0
1010
#
1111
# Unless required by applicable law or agreed to in writing, software
1212
# distributed under the License is distributed on an "AS IS" BASIS,
@@ -28,4 +28,4 @@ fi
2828
export CLASSPATH="$(find target/ -type f -name '*.jar'| grep '\-package' | tr '\n' ':')"
2929
export KAFKA_JMX_OPTS='-Xdebug -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005'
3030

31-
$CONFLUENT_HOME/bin/connect-standalone connect/connect-avro-docker.properties config/SplunkHttpSink.properties
31+
$CONFLUENT_HOME/bin/connect-standalone connect/connect-avro-docker.properties config/SplunkHttpSource.properties config/SplunkHttpSink.properties

‎bin/suspend.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
#!/usr/bin/env bash
22
#
3-
# Copyright (C) ${project.inceptionYear} Jeremy Custenborder ([email protected])
3+
# Copyright © 2016 Jeremy Custenborder ([email protected])
44
#
55
# Licensed under the Apache License, Version 2.0 (the "License");
66
# you may not use this file except in compliance with the License.
77
# You may obtain a copy of the License at
88
#
9-
# http://www.apache.org/licenses/LICENSE-2.0
9+
# http://www.apache.org/licenses/LICENSE-2.0
1010
#
1111
# Unless required by applicable law or agreed to in writing, software
1212
# distributed under the License is distributed on an "AS IS" BASIS,
@@ -28,4 +28,4 @@ fi
2828
export CLASSPATH="$(find target/ -type f -name '*.jar'| grep '\-package' | tr '\n' ':')"
2929
export KAFKA_JMX_OPTS='-Xdebug -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005'
3030

31-
$CONFLUENT_HOME/bin/connect-standalone connect/connect-avro-docker.properties config/salesforce.properties
31+
$CONFLUENT_HOME/bin/connect-standalone connect/connect-avro-docker.properties config/SplunkHttpSink.properties config/SplunkHttpSource.properties

‎checkstyle/checkstyle.xml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<!--
33
4-
Copyright (C) ${project.inceptionYear} Jeremy Custenborder ([email protected])
4+
Copyright © 2016 Jeremy Custenborder ([email protected])
55
66
Licensed under the Apache License, Version 2.0 (the "License");
77
you may not use this file except in compliance with the License.
88
You may obtain a copy of the License at
99
10-
http://www.apache.org/licenses/LICENSE-2.0
10+
http://www.apache.org/licenses/LICENSE-2.0
1111
1212
Unless required by applicable law or agreed to in writing, software
1313
distributed under the License is distributed on an "AS IS" BASIS,

‎config/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
SplunkHttpSource.properties

‎config/SplunkHttpSink.properties‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
#
2-
# Copyright (C) ${project.inceptionYear} Jeremy Custenborder ([email protected])
2+
# Copyright © 2016 Jeremy Custenborder ([email protected])
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.
66
# You may obtain a copy of the License at
77
#
8-
# http://www.apache.org/licenses/LICENSE-2.0
8+
# http://www.apache.org/licenses/LICENSE-2.0
99
#
1010
# Unless required by applicable law or agreed to in writing, software
1111
# distributed under the License is distributed on an "AS IS" BASIS,

‎config/MySourceConnector.properties‎ renamed to ‎config/SplunkHttpSource.example.properties‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
#
2-
# Copyright (C) ${project.inceptionYear} Jeremy Custenborder ([email protected])
2+
# Copyright © 2016 Jeremy Custenborder ([email protected])
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.
66
# You may obtain a copy of the License at
77
#
8-
# http://www.apache.org/licenses/LICENSE-2.0
8+
# http://www.apache.org/licenses/LICENSE-2.0
99
#
1010
# Unless required by applicable law or agreed to in writing, software
1111
# distributed under the License is distributed on an "AS IS" BASIS,
@@ -14,6 +14,8 @@
1414
# limitations under the License.
1515
#
1616

17-
name=MySourceConnector
17+
name=splunk-http-source
1818
tasks.max=1
19-
connector.class=io.confluent.kafka.connect.splunk.SplunkHttpSourceConnector
19+
connector.class=io.confluent.kafka.connect.splunk.SplunkHttpSourceConnector
20+
splunk.ssl.key.store.path=
21+
splunk.ssl.key.store.password=

‎config/SplunkHttpSource.properties‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#
2+
# Copyright © 2016 Jeremy Custenborder ([email protected])
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
name=splunk-http-source
18+
tasks.max=1
19+
connector.class=io.confluent.kafka.connect.splunk.SplunkHttpSourceConnector
20+
splunk.ssl.key.store.path=/Users/jeremy/keystores/keystore.jks
21+
splunk.ssl.key.store.password=password
22+
splunk.collector.index.default=main

‎connect/connect-avro-docker.properties‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
#
2-
# Copyright (C) ${project.inceptionYear} Jeremy Custenborder ([email protected])
2+
# Copyright © 2016 Jeremy Custenborder ([email protected])
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.
66
# You may obtain a copy of the License at
77
#
8-
# http://www.apache.org/licenses/LICENSE-2.0
8+
# http://www.apache.org/licenses/LICENSE-2.0
99
#
1010
# Unless required by applicable law or agreed to in writing, software
1111
# distributed under the License is distributed on an "AS IS" BASIS,
@@ -39,3 +39,4 @@ offset.storage.file.filename=/tmp/connect.offsets
3939
# that will report audit data that can be displayed and analyzed in Confluent Control Center
4040
# producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor
4141
# consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor
42+
consumer.max.poll.records=500

‎docker-compose.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
#
2-
# Copyright (C) ${project.inceptionYear} Jeremy Custenborder ([email protected])
2+
# Copyright © 2016 Jeremy Custenborder ([email protected])
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.
66
# You may obtain a copy of the License at
77
#
8-
# http://www.apache.org/licenses/LICENSE-2.0
8+
# http://www.apache.org/licenses/LICENSE-2.0
99
#
1010
# Unless required by applicable law or agreed to in writing, software
1111
# distributed under the License is distributed on an "AS IS" BASIS,

0 commit comments

Comments
 (0)