Skip to content
This repository was archived by the owner on Sep 2, 2025. It is now read-only.

INTEXT-72: Implement realtime search - #1

Open
artembilan wants to merge 7 commits into
spring-attic:masterfrom
artembilan:INTEXT-72
Open

INTEXT-72: Implement realtime search#1
artembilan wants to merge 7 commits into
spring-attic:masterfrom
artembilan:INTEXT-72

Conversation

@artembilan

Copy link
Copy Markdown
Contributor

JIRA: https://jira.spring.io/browse/INTEXT-72

An initial polishing commit.

There is no yet real implementation

JIRA: https://jira.spring.io/browse/INTEXT-72

An initial polishing commit.

There is no yet real implementation
@artembilan

Copy link
Copy Markdown
Contributor Author

@dturanski , would you mind answering or at least helpping who can?
The Splunk Job has states: isDone(), isFailed(), isFinalized(), isPaused(). How are they related to each other? The Job JavaDocs don't provide good info and I can't find an answer in the Internet.

In most cases we do:

while (!job.isDone()) {
    Thread.sleep(2000);
}

However I think we may don't get isDone status, if job has been falied or finalized manually from Splunk.
WDYT?

When I will have had an answer, I'll know how to go ahead with realtime search correctly.
Because this type of Job is never done and it is normal: http://dev.splunk.com/view/java-sdk/SP-CAAAEHQ.

* Add `SplunkDataReader#realtimeSearch`
* Add `SplunkRunning` test Rule
* Add Real test-case - `RealTimeSearchIntegrationTests`
* Allow empty string for `init-earliest-time` of `<int-splunk:inbound-channel-adapter>`. For realtime search it is interpreted as
`earliest_time = "rt"` and `latest_time="rt"`. However `sliding window` is still valid option for the realtime search as well.
* Polishing for configs
* Polishing `README.md` and add a note regarding realtime search
@artembilan

Copy link
Copy Markdown
Contributor Author

Pushed the implementation for realtime search.

Pay attention, please, to the SplunkDataReader#isJobInvalid() method.
Testing with real Splunk Enterprise shows that finalized and failed states impact to the done one and we always get true for those cases as well.
However there is a case when we get an Exception from Job.refresh(), e.g. when we manually remove the Job from Splunk Enterprise - SID isn't valid.
For all other searches it is OK. We end up with an Exception which is propagated as an ErrorMessage to the error-channel from the <poller> and the next poll initiates a new Job.
For the realtime search we always should have only the single long-living Job. Hance we should schedule a new one, when we get an exception from Splunk Service.

Hope I am clear.

Some additional fixes:
* reset `realTimeSearchResultOffset = 0`, when we restart RT Job
* Fix `SplunkOutboundChannelAdapterParser` to use just `commaDelimitedListToStringArray` for the `splunk-server-ref` list
instead of `;` delimiter before
@artembilan

Copy link
Copy Markdown
Contributor Author

Since RT search Job is long-living but we can start/stop adapter at runtime, consider to finish() Splunk Job on adapter stop().
See https://jira.spring.io/browse/INT-3571

@artembilan

Copy link
Copy Markdown
Contributor Author

Pushed Lifecycle for RT search Jobs based on the latests SI 4.0.6 changes

@artembilan

Copy link
Copy Markdown
Contributor Author

Pushed isFalied() check

@artembilan

Copy link
Copy Markdown
Contributor Author

Pushed upgrade commit

@artembilan

Copy link
Copy Markdown
Contributor Author

Pushed upgrade and IO-2.0 compatibility.

CC @wilkinsona

@wilkinsona

Copy link
Copy Markdown
Contributor

Thanks for taking care of the plugin upgrade, @artembilan. One less PR for me to do :)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants