本文介绍了Logstash:无法连接到外部Amazon RDS数据库的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我对logstash&;Elasticsearch相对较新.
已在MacOS Mojave上安装使用的logstash&;Elasticsearch(10.14.2):
brew install logstash brew install elasticsearch当我检查这些版本时:
brew list --versions接收以下输出:
elasticsearch 6.5.4 logstash 6.5.4当我打开Google Chrome并在URL地址字段中键入以下内容时:
localhost:9200这是我收到的JSON响应:
{ "name" : "9oJAP16", "cluster_name" : "elasticsearch_local", "cluster_uuid" : "PgaDRw8rSJi-NDo80v_6gQ", "version" : { "number" : "6.5.4", "build_flavor" : "oss", "build_type" : "tar", "build_hash" : "d2ef93d", "build_date" : "2018-12-17T21:17:40.758843Z", "build_snapshot" : false, "lucene_version" : "7.5.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" }内部:
/usr/local/etc/logstash/logstash.yml驻留以下变量:
path.data: /usr/local/Cellar/logstash/6.5.4/libexec/data pipeline.workers: 2 path.config: /usr/local/etc/logstash/conf.d log.level: info path.logs: /usr/local/var/log内部:
/usr/local/etc/logstash/pipelines.yml驻留以下变量:
- pipeline.id: main path.config: "/usr/local/etc/logstash/conf.d/*.conf"在下面设置了以下logstash_etl.conf文件:
/usr/local/etc/logstash/conf.d内容:
input { jdbc { jdbc_connection_string => "jdbc:mysql://myapp-production.crankbftdpmc.us-west-2.rds.amazonaws:3306/products" jdbc_user => "products_admin" jdbc_password => "products123" jdbc_driver_library => "/etc/logstash/mysql-connector/mysql-connector-java-5.1.21.jar" jdbc_driver_class => "com.mysql.jdbc.driver" schedule => "*/5 * * * *" statement => "select * from products" use_column_value => false clean_run => true } } # sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-exec output { if ([purge_task] == "yes") { exec { command => "curl -XPOST 'localhost:9200/_all/products/_delete_by_query?conflicts=proceed' -H 'Content-Type: application/json' -d' { "query": { "range" : { "@timestamp" : { "lte" : "now-3h" } } } } '" } } else { stdout { codec => json_lines} elasticsearch { "hosts" => "localhost:9200" "index" => "product_%{product_api_key}" "document_type" => "%{[@metadata][index_type]}" "document_id" => "%{[@metadata][index_id]}" "doc_as_upsert" => true "action" => "update" "retry_on_conflict" => 7 } } }执行此操作时:
brew services start logstash在我的/usr/local/var/log/logstash-Pla.log文件中收到以下内容:
[2019-01-15T14:51:15,319][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x399927c7 run>"} [2019-01-15T14:51:15,663][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2019-01-15T14:51:16,514][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2019-01-15T14:57:31,432][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Tried 1 times {:error_message=>"Java::ComMysqlCjJdbcExceptions::CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."} [2019-01-15T14:57:31,435][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Tried 1 times {:error_message=>"Java::ComMysqlCjJdbcExceptions::CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}[2019-01-15T14:51:15,319][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x399927c7 run>"} [2019-01-15T14:51:15,663][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2019-01-15T14:51:16,514][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2019-01-15T14:57:31,432][ERROR][logstash.inputs.jdbc ] Unable to connect to database. Tried 1 times我可能做错了什么?
有没有办法从Elasticsearch服务器(暂存或生产)获取转储(例如mysqldump),然后重新导入到运行Elasticsearch的本地实例,而不使用logstash?
这与在Amazon EC-2生产实例中工作的配置文件相同,但不知道为什么它不能在我的本地MacOS Mojave实例中工作?
rds您可能会遇到推荐答案的ssl问题,因为
如果您使用MySQL Java Connector v5.1.38或更高版本,或者MySQL Java Connector v8.0.9或更高版本连接到数据库,即使您没有将应用程序显式配置为在连接到数据库时使用SSL/TLS,这些客户端驱动程序也会默认使用SSL/TLS。此外,在使用SSL/TLS时,它们执行部分证书验证,如果数据库服务器证书过期,则连接失败。如AWS RDS Doc
中所述若要克服此问题,请为LogStash设置信任存储区,上面的链接也介绍了这一点。
或冒险禁用连接字符串中的SSL,如
jdbc_connection_string => "jdbc:mysql://myapp-production.crankbftdpmc.us-west-2.rds.amazonaws:3306/products?sslMode=DISABLED"更多推荐
Logstash:无法连接到外部Amazon RDS数据库
发布评论