再见 Fluentd,你好 Fluent Bit

请原谅我起了这么一个标题党的名字。之前用 Fluentd 搭建过日志收集系统(参见:基于 fluentd 搭建日志收集系统),所以当再次需要搭建日志系统的时候,第一时间就想到了安利一波 Fluentd。结果在浏览官网的时候看到了 Fluent Bit,果断就被安利了。

Fluentd vs Fluent Bit

作为同一家公司出品的日志收集软件,Fluent Bit 自诞生起就不可避免的被拿来和 Fluentd 进行比较。首先说结论,我认为 Fluent Bit 并不是用来替代 Fluentd 的。我们不妨通过官网的介绍来看他们各自的定位:Fluentd 作为一个全功能的 Data Collector,而 Fluent Bit 是作为一个轻量级的 Data Forwarder。

Fluentd is an open source data collector for unified logging layer. Fluentd allows you to unify data collection and consumption for a better use and understanding of data.

Fluent Bit is a Fast and Lightweight Data Forwarder for Linux, OSX and BSD family operating systems. It has been made with a strong focus on performance to allow the collection of events from different sources without complexity.

Fluentd Fluent Bit
Scope Containers / Servers Containers / Servers
Language C & Ruby C
Memory ~40MB ~450KB
Performance High Performance High Performance
Dependencies Built as a Ruby Gem, it requires a certain number of gems. Zero dependencies, unless some special plugin requires them.
Plugins More than 650 plugins available Around 35 plugins available
License Apache License v2.0 Apache License v2.0

参考:

http://fluentbit.io/

https://www.fluentd.org/

925 total views, no views today

基于 fluentd 搭建日志收集系统

测试系统版本为 ubuntu 14.04,fluentd 版本为 td-agent 2.2.1。使用 HTTP 方式接收日志数据,并存储到 mysql 仓库。

安装 fluentd

1. 去官网下载deb安装包
2. dpkg -i td-agent_2.2.1-0_amd64.deb
3. 安装好之后td-agent会自启动
4. 测试一下,curl -XPOST 192.168.1.107:8888
400 Bad Request
'json' or 'msgpack' parameter is required
说明工作正常。

/etc/init.d/td-agent status
* td-agent is running

安装 fluentd-ui

1. sudo /usr/sbin/td-agent-ui start
2. 默认端口9292,用户名admin,密码changeme

安装 fluent-plugin-mysql

1. sudo apt-get install libmysqlclient-dev
2. /usr/sbin/td-agent-gem sources --add https://ruby.taobao.org/ --remove https://rubygems.org/
3. /usr/sbin/td-agent-gem install fluent-plugin-mysql


参考:

http://www.fluentd.org

http://stackoverflow.com/questions/3608287/error-installing-mysql2-failed-to-build-gem-native-extension

590 total views, no views today