As mentioned at #4043 (comment), it would be great to have a way to opt-out of thinking about some of the configuration parameters for the file_source::FileServer. However, it currently doesn't implement the Default trait.
However, there is a default implementation at the vector::sources::file::FileConfig with a lot of useful defaults.
There are some usability problems we'd like to solve here:
- one could construct the
Default vector::sources::file::FileConfig and take some value from it, but it's suboptimal: in advanced use cases, vector::sources::file::FileConfig parameters don't directly translate to the file_source::FileServer (like when using a custom path_provider);
vector::sources::file::FileConfig is scoped to the vector::sources::file, while we might want to use it in other source, i.e. vector::sources::kubernetes_logs;
- specifying all the parameters inline at each non-standard
file_source::FileServer initialization is problematic because then there's no distinction between the parameters that are set to their respective values explicitly or just set to whatever the defaults are presumed to be; as already mentioned, we'd like to have a way to just opt-out of specifying the default values, omit the fields and rely on default-initialization.
To achieve it, the proposed solution is to alter the file_source::FileServer in such a way that it is constructible via some other struct, that has the Default-initializer (with all the "primitve" or trivial values in it), and a set of other parameters that are hard to provide defaults for (generic or otherwise abstracted):
pub struct FileServer<PP, E: FileSourceInternalEvents>
where
PP: PathsProvider,
{
pub paths_provider: PP,
pub emitter: E,
pub config: Config,
}
pub struct Config {
pub max_read_bytes: usize,
pub start_at_beginning: bool,
pub ignore_before: Option<time::SystemTime>,
pub max_line_bytes: usize,
pub data_dir: PathBuf,
pub glob_minimum_cooldown: Duration,
pub fingerprinter: Fingerprinter,
pub oldest_first: bool,
pub remove_after: Option<Duration>,
}
impl Default for Config {
// ...
}
This is just a sample code to illustrate the idea, the exact structure is worth proper discussion.
@lukesteensen, do you like this approach?
As mentioned at #4043 (comment), it would be great to have a way to opt-out of thinking about some of the configuration parameters for the
file_source::FileServer. However, it currently doesn't implement theDefaulttrait.However, there is a default implementation at the
vector::sources::file::FileConfigwith a lot of useful defaults.There are some usability problems we'd like to solve here:
Defaultvector::sources::file::FileConfigand take some value from it, but it's suboptimal: in advanced use cases,vector::sources::file::FileConfigparameters don't directly translate to thefile_source::FileServer(like when using a custompath_provider);vector::sources::file::FileConfigis scoped to thevector::sources::file, while we might want to use it in other source, i.e.vector::sources::kubernetes_logs;file_source::FileServerinitialization is problematic because then there's no distinction between the parameters that are set to their respective values explicitly or just set to whatever the defaults are presumed to be; as already mentioned, we'd like to have a way to just opt-out of specifying the default values, omit the fields and rely on default-initialization.To achieve it, the proposed solution is to alter the
file_source::FileServerin such a way that it is constructible via some other struct, that has theDefault-initializer (with all the "primitve" or trivial values in it), and a set of other parameters that are hard to provide defaults for (generic or otherwise abstracted):This is just a sample code to illustrate the idea, the exact structure is worth proper discussion.
@lukesteensen, do you like this approach?