[Enhancement] Define and standardize some common configurations for all Sources #5101
Closed
3 tasks done
Labels
enhancement
New feature or request
Search before asking
Enhancement Request
In the various implementations of
Source
, there are often some common configurations scattered across different places, such as the capacity of the storage queue, the number of elements to poll each time, etc. Additionally, in some source implementations, thepoll
method can block for up to several seconds in the worst-case scenario, reachingmaxBatchSize * pollTimeout
. Therefore, it is necessary to define common parameters that support configuration and also limit the maximum waiting time forpoll
.Describe the solution you'd like
PollConfig
configuration class that includes parameters forcapacity
,maxBatchSize
, andmaxWaitTime
, and incorporated it as a variable inSourceConfig
to support user configuration with default values.capacity
,maxBatchSize
, andmaxWaitTime
in the original source implementations with the corresponding variables fromPollConfig
.poll
methods to ensure the maximum waiting time does not exceedmaxWaitTime
, preventing blocking for several seconds.Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: