Tuning parameter to consider:Ī)Tiered data collection flows : It help to distribute load and scale-able architecture.ī)Type of channel:File ,Memory. The channel selector property can be used for configuring these flows.Ĭ)Tiered data collection flow: These flows can be used for configuring multiple Flume agents such that they receive data from the initial sources and consolidate the data onto fewer agents that can finally dump the data into the final sink.It is used in log collection. These can be either multiplexing or replicating in nature. Multiple agents can be connected in a series-like configuration, wherein the sink of one agent is connected to the source of another agent.ī)Fan-Out flow: In this type flows, multiple channels are connected to the same source. These flows are preferred when the rate at which data is generated is high. It represents the path taken by the data from its source to reach the target destination using Flume agents.Ī)Multi-agent flow: In these flows, more than one flume agents are used. The generic template of a Flume configuration file #list sources, sinks and channels in the agent Recommendation systems, sentiment analysis using Twitter It is part of flume agent which deliver data to final destination.It work on Pull method and once data is written in destination it inform channel to remove that events.It uses a transactional approach to guarantee the reliable delivery of the events.The sources and sinks encapsulate in a transaction the storage/retrieval, respectively, of the events placed in or provided by a transaction provided by the channel. It is buffer that keeps events until sink write to storage target.Multiple source can write to channel and multiple sink can read events from same channel.It support JDBC and kafka channel,priority tracking of events.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |