环境"/>
在Mac上使用idea搭建flink java开发环境
1.环境
本文档记录的是使用flink的java API简单地创建应用的过程。
前置条件:需要安装Java、maven和flink。
1.1 Java环境变量
Java需要jdk、path、classpath等环境变量,这里使用Mac下自带的jdk,配置如下:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home
PATH=$JAVA_HOME/bin:$PATH:.
CLASSPATH=$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:.
export JAVA_HOME
export PATH
export CLASSPATH
通过在终端运行java -version命令,如果成功输出Java对版本,说明Java环境变量配置成功。
1.2 maven环境变量配置
注意:在安装maven之前你需要提前安装Java运行环境。在安装maven的时候,需要将maven及maven bin所在的目录添加到环境变量中,并将maven bin添加到PATH中。
export MAVEN_HOME=/Users/xxx/DevTools/apache-maven-3.6.1
export MAVEN_BIN=$MAVEN_HOME/bin
export PATH=$MAVEN_BIN:$PATH
安装完maven后,输入mvn -v命令验证使用安装成功,成功后会出现如下所示的信息:
$ mvn -v
Apache Maven 3.6.1 (2019-04-05T03:00:29+08:00)
Maven home: /Users/xxx/DevTools/apache-maven-3.6.1
Java version: 1.8.0_201, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk1.8.0_201.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.2", arch: "x86_64", family: "mac"
1.3 flink环境配置
安装完Java、maven后,可以进行flink的安装和配置。
1.3.1 flink安装
如果安装过brew,那么在 Mac OS X 上安装 Flink 是比较方便的,这里在Mac环境下推荐通过 homebrew 对flink进行安装:
brew install apache-flink
使用以上命令,默认将flink安装在/usr/local/Cellar/apache-flink/目录中。
1.3.2 检查安装是否成功
通过运行flink --version命令检查flink是否安装成功。
MacBook-Pro:~ sym$ flink --version
Version: 1.8.1, Commit ID: 7303dbe......
1.3.3 启动flink
/usr/local/Cellar/apache-flink/1.8.1/libexec/bin ./start-cluster.sh
Starting cluster.
Starting standalonesession daemon on host sym.
Starting taskexecutor daemon on host sym.
2.Mac上使用idea搭建flink java开发环境
这里使用 maven archetypes 命令的方式来构建项目。
2.1 进入到要创建项目的工作目录
例如,我想创建在 IdeaProjects /Users/shenym/DevTools/apache-maven-3.6.1/repository/下,进入该目录
$ cd /Users/shenym/DevTools/apache-maven-3.6.1/repository/
2.2 使用 Maven 创建项目
1.打开 Terminal,键入
$ mvn archetype:generate
2.回车,会出现如下信息
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:3.1.1:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:3.1.10:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:3.1.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
3.回车,当出现Define value for property 'groupId’时,键入如下内容,并回车:
Define value for property 'groupId': org.sym.flink
4.当出现如下等Define value for property ‘artifactId’:和 Define value for property ‘version’ 1.0-SNAPSHOT:信息时,键入如下,并回车:
Define value for property 'groupId': com.sym.flink
Define value for property 'artifactId': FlinkTest
Define value for property 'version' 1.0-SNAPSHOT: :
Define value for property 'package' com.sym.flink: :
Confirm properties configuration:
groupId: com.sym.flink
artifactId: FlinkTest
version: 1.0-SNAPSHOT
package: com.sym.flinkY: : [INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: maven-archetype-quickstart:1.4
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.sym.flink
[INFO] Parameter: artifactId, Value: FlinkTest
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: com.sym.flink
[INFO] Parameter: packageInPathFormat, Value: com/sym/flink
[INFO] Parameter: package, Value: com.sym.flink
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: groupId, Value: com.sym.flink
[INFO] Parameter: artifactId, Value: FlinkTest
[INFO] Project created from Archetype in dir: /Users/shenym/DevTools/apache-maven-3.6.1/repository/FlinkTest
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30:36 min
[INFO] Finished at: 2019-07-28T14:12:14+08:00
[INFO] ------------------------------------------------------------------------
5.出现 BUILD SUCCESS 信息时,表示创建成功。
6.使用idea打开生成的FlinkTest项目后,该项目的目录结构如下图:
2.3 使用 IntelliJ IDEA 打开创建好的项目
1.打开 Intellij IDEA,选择Import Project(如果有使用IDEA打开的项目,需要退出重新打开才会有Import Project选项)
2.选择 Import project from external model,选择Maven,点击Next
3.使用默认选项就好,点击Next
4.点击Next
5.继续点击Next
6.点击 Finish
7.导入成功,即可看到如下图所示的项目
8.打开pom.xml,添加依赖
参考其官方链接.8/dev/projectsetup/dependencies.html,导入 Java 编程环境必须的依赖。本来使用最新的Flink-1.8.1,但是发现未能导入成功,就换成下面的依赖。官网提供了 Java 和 Scala的,按自己需要添加。下面是 在pom.xml中导入的Java 环境的必需依赖:
<dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.11</version><scope>test</scope></dependency><!-- .apache.flink/flink-java --><dependency><groupId>org.apache.flink</groupId><artifactId>flink-java</artifactId><version>1.7.2</version></dependency><!-- .apache.flink/flink-streaming-java --><dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-java_2.12</artifactId><version>1.7.2</version><scope>provided</scope></dependency></dependencies>
将上面这部分代码复制到 pom.xml 的里面复制完后,点击右下角的"Enable Auto-import"。
9.配置Project Structure的Modules模型
File → Project Stucture → Modules中进行如下配置,以导入下载到flink中的 lib 和 opt 包:
(1)打开 Project Stucture;(2)选择左列中的 Modules;(3)选择右列中的 Dependencies;(4)选择右列中左下角的"+"号;(5)选择"1 JARs or directories";(6)选择下载的 flink-1.8.1/libexec 中的 lib 和 opt 包;
(7)点击 apply 和 ok。
(8)再次查看程序,发现import org.apache.flink包成功,即可在idea中使用maven的方式编写flink java程序。
3.创建项目
WordCount 测试代码
原文:
使用下面其中一个命令来创建Flink Java工程:
3.1 创建项目命令
3.1.1 使用maven archetype命令
$ mvn archetype:generate \-DarchetypeGroupId=org.apache.flink \-DarchetypeArtifactId=flink-quickstart-java \-DarchetypeCatalog=/ \-DarchetypeVersion=1.3-SNAPSHOT
3.1.2 运行quickstart脚本
$ curl .sh | bash
这里使用quickstart脚本的方式,创建flink java工程,运行上面命令后,出现如下所示信息:
MacBook-Pro:~ xxx$ curl .sh | bash% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 2045 100 2045 0 0 484 0 0:00:04 0:00:04 --:--:-- 484
[INFO] Scanning for projects...
Downloading from central: .4/maven-archetype-plugin-2.4.pom
Downloaded from central: .4/maven-archetype-plugin-2.4.pom (8.4 kB at 4.6 kB/s)
Downloading from central: .4/maven-archetype-2.4.pom
Downloaded from central: .4/maven-archetype-2.4.pom (13 kB at 7.5 kB/s)
Downloading from central: .pom
Downloaded from central: .pom (40 kB at 11 kB/s)
Downloading from central: .pom
Downloaded from central: .pom (15 kB at 9.0 kB/s)
Downloading from central: .4/maven-archetype-plugin-2.4.jar
Downloaded from central: .4/maven-archetype-plugin-2.4.jar (93 kB at 19 kB/s)
[INFO]
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:2.4:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:2.4:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO]
[INFO]
[INFO] --- maven-archetype-plugin:2.4:generate (default-cli) @ standalone-pom ---
Downloading from central: .4/archetype-catalog-2.4.pom
Downloaded from central: .4/archetype-catalog-2.4.pom (1.9 kB at 4.9 kB/s)
Downloading from central: .4/archetype-models-2.4.pom
Downloaded from central: .4/archetype-models-2.4.pom (2.8 kB at 3.5 kB/s)
Downloading from central: .0.21/plexus-utils-3.0.21.pom
Downloaded from central: .0.21/plexus-utils-3.0.21.pom (3.8 kB at 3.8 kB/s)
Downloading from central: .3.1/plexus-3.3.1.pom
Downloaded from central: .3.1/plexus-3.3.1.pom (20 kB at 11 kB/s)
Downloading from central: .pom
Downloaded from central: .pom (6.8 kB at 8.3 kB/s)
Downloading from central: .pom
Downloaded from central: .pom (14 kB at 11 kB/s)
Downloading from central: .4/archetype-descriptor-2.4.pom
Downloaded from central: .4/archetype-descriptor-2.4.pom (1.9 kB at 4.6 kB/s)
Downloading from central: .4/archetype-registry-2.4.pom
Downloaded from central: .4/archetype-registry-2.4.pom (1.9 kB at 1.6 kB/s)
Downloading from central: .4/archetype-common-2.4.pom
Downloaded from central: .4/archetype-common-2.4.pom (16 kB at 3.1 kB/s)
Downloading from central: .0/jchardet-1.0.pom
Downloaded from central: .0/jchardet-1.0.pom (1.3 kB at 3.4 kB/s)
Downloading from central: .5.5/plexus-component-annotations-1.5.5.pom
Downloaded from central: .5.5/plexus-component-annotations-1.5.5.pom (815 B at 795 B/s)
Downloading from central: .5.5/plexus-containers-1.5.5.pom
Downloaded from central: .5.5/plexus-containers-1.5.5.pom (4.2 kB at 7.2 kB/s)
Downloading from central: .0.7/plexus-2.0.7.pom
Downloaded from central: .0.7/plexus-2.0.7.pom (17 kB at 10 kB/s)
Downloading from central: .6.1/dom4j-1.6.1.pom
Downloaded from central: .6.1/dom4j-1.6.1.pom (6.8 kB at 3.3 kB/s)
Downloading from central: .0.b2/xml-apis-1.0.b2.pom
Downloaded from central: .0.b2/xml-apis-1.0.b2.pom (2.2 kB at 1.8 kB/s)
Downloading from central: .0/jdom-1.0.pom
Downloaded from central: .0/jdom-1.0.pom (1.2 kB at 2.4 kB/s)
Downloading from central: .2.1/maven-model-2.2.1.pom
Progress (1): 3.2 kB
......
3.2 创建maven工程
参考:
修改maven的conf/setting.xml文件,并配置本地仓库可以参考
注意:由于maven使用的是自己下载的,而且配置了环境变量,所以在创建maven工程的时候,不要使用idea自带的,否则容易出错。
启动命令:
MacBook-Pro:bin sym$ flink run -c com.sym.flink.wordcount.SocketTextStreamWordCount /Users/shenym/DevTools/apache-maven-3.6.1/repository/FlinkTest/target/FlinkTest-1.0-SNAPSHOT.jar 127.0.0.1 9000Starting execution of program
4.Flink基本概念和原理
参考:
5.遇到的问题
5.1 idea 报错maven [ERROR] Maven execution terminated abnormally (exit code 1)
参考:
5.2 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.1.1:generate (default-cli) on project standalone-pom
参考:
1.首先进入仓库下面repository\org\apache\maven\plugins这个目录;
2.删除目录下的maven-archetype-plugin文件夹;
3.在maven安装目录使用命令,重新加载:mvn archetype:generate,执行该命令后在命令行中出现如下信息说明构建maven项目成功;
Define value for property 'groupId': com.sym.flink
Define value for property 'artifactId': FlinkTest
Define value for property 'version' 1.0-SNAPSHOT: :
Define value for property 'package' com.sym.flink: :
Confirm properties configuration:
groupId: com.sym.flink
artifactId: FlinkTest
version: 1.0-SNAPSHOT
package: com.sym.flinkY: :
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: maven-archetype-quickstart:1.4
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.sym.flink
[INFO] Parameter: artifactId, Value: FlinkTest
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: com.sym.flink
[INFO] Parameter: packageInPathFormat, Value: com/sym/flink
[INFO] Parameter: package, Value: com.sym.flink
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: groupId, Value: com.sym.flink
[INFO] Parameter: artifactId, Value: FlinkTest
[INFO] Project created from Archetype in dir: /Users/sym/DevTools/apache-maven-3.6.1/repository/FlinkTest
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30:36 min
[INFO] Finished at: 2019-07-28T14:12:14+08:00
4.在idea中导入刚构建的工程:File → Open → 项目所在路径(这里为/Users/sym/DevTools/apache-maven-3.6.1/repository/FlinkTest),打开后项目目录如图。
5.3 创建完maven项目,在pom.xml中导入flink的如下依赖时,发现无法导入(该包显示为红色)
<dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.11</version><scope>test</scope></dependency><!-- .apache.flink/flink-java --><dependency><groupId>org.apache.flink</groupId><artifactId>flink-java</artifactId><version>1.7.2</version></dependency><!-- .apache.flink/flink-streaming-java --><dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-java_2.12</artifactId><version>1.7.2</version><scope>provided</scope></dependency></dependencies>
解决方式:
File → Project Stucture → Modules中进行如下配置,以导入下载到flink中的 lib 和 opt 包:
打开 Project Stucture;
选择左列中的 Modules;
选择右列中的 Dependencies;
选择右列中左下角的"+“号;
选择"1 JARs or directories”;
6.选择下载的 flink-1.8.1/libexec 中的 lib 和 opt 包;
7.点击 apply 和 ok。
8.再次查看程序,发现import org.apache.flink包成功。
注意:flink默认安装在/usr/local/Cellar/apache-flink目录中,而在mac中/usr/local/目录默认是隐藏不可见的,好在Mac OS可以通过快捷键Command + Shift + . 来快速(在 Finder 中)显示和隐藏隐藏文件。
参考:
MacOS 在 Intellij IDEA 上配置 Flink-1.8.0 环境:
更多推荐
在Mac上使用idea搭建flink java开发环境
发布评论