1. 使用官方快速创建一个项目快速创建项目目的是减少在开发过程中解决包依赖的关系出现错误和版本冲突的问题无法调度的问题浪费大量的时间官方的网址如下https://nightlies.apache.org/flink/flink-docs-release-1.20/zh/docs/dev/configuration/overview/windowsliunx 项目目录下执行如下命令后面跟上版本mvn archetype:generate -DarchetypeGroupIdorg.apache.flink -DarchetypeArtifactIdflink-quickstart-java -DarchetypeVersion1.20.3curl https://flink.apache.org/q/quickstart.sh | bash -s 1.20.32. 根据自已项目的需求开发和引入相关的jar,如下所示一个实例的pom文件!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -- project xmlnshttp://maven.apache.org/POM/4.0.0 xmlns:xsihttp://www.w3.org/2001/XMLSchema-instance xsi:schemaLocationhttp://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd modelVersion4.0.0/modelVersion groupIdorg/groupId artifactIdexample/artifactId version1.0/version packagingjar/packaging nameFlink Quickstart Job/name properties project.build.sourceEncodingUTF-8/project.build.sourceEncoding flink.version1.20.2/flink.version target.java.version1.8/target.java.version scala.binary.version2.12/scala.binary.version maven.compiler.source${target.java.version}/maven.compiler.source maven.compiler.target${target.java.version}/maven.compiler.target log4j.version2.24.3/log4j.version jedis.version3.0.1/jedis.version /properties repositories repository idapache.snapshots/id nameApache Development Snapshot Repository/name urlhttps://repository.apache.org/content/repositories/snapshots//url releases enabledfalse/enabled /releases snapshots enabledtrue/enabled /snapshots /repository /repositories dependencies !-- Apache Flink dependencies -- !-- These dependencies are provided, because they should not be packaged into the JAR file. -- dependency groupIdorg.apache.flink/groupId artifactIdflink-streaming-java/artifactId version${flink.version}/version scopeprovided/scope /dependency dependency groupIdorg.apache.flink/groupId artifactIdflink-clients/artifactId version${flink.version}/version scopeprovided/scope /dependency !-- Add connector dependencies here. They must be in the default scope (compile). -- !-- Example: dependency groupIdorg.apache.flink/groupId artifactIdflink-connector-kafka/artifactId version3.0.0-1.17/version /dependency -- !-- Add logging framework, to produce console output when running in the IDE. -- !-- These dependencies are excluded from the application JAR by default. -- dependency groupIdorg.apache.logging.log4j/groupId artifactIdlog4j-slf4j-impl/artifactId version${log4j.version}/version scoperuntime/scope /dependency dependency groupIdorg.apache.logging.log4j/groupId artifactIdlog4j-api/artifactId version${log4j.version}/version scoperuntime/scope /dependency dependency groupIdorg.apache.logging.log4j/groupId artifactIdlog4j-core/artifactId version${log4j.version}/version scoperuntime/scope /dependency !-- 自行添加的内容 -- !-- Flink Kafka 连接器 -- dependency groupIdorg.apache.flink/groupId artifactIdflink-connector-kafka/artifactId version3.2.0-1.18/version /dependency !-- Flink JDBC 连接器 (核心依赖) -- dependency groupIdorg.apache.flink/groupId artifactIdflink-connector-jdbc/artifactId version3.2.0-1.18/version /dependency dependency groupIdorg.apache.flink/groupId artifactIdflink-connector-base/artifactId version1.20.2/version /dependency !--Flink 官方推荐用 maven-shade-plugin 或 flink-assembly-plugin 统一打包-- dependency groupIdorg.apache.flink/groupId artifactIdflink-table-api-java-bridge/artifactId version1.20.2/version !-- 替换为你的 Flink 版本 确保版本统一管理 -- exclusions exclusion artifactIdflink-java/artifactId groupIdorg.apache.flink/groupId /exclusion /exclusions /dependency !-- Flink Table Planner必须添加 -- dependency groupIdorg.apache.flink/groupId artifactIdflink-table-planner_2.12/artifactId version1.20.2/version /dependency !-- flink-table-runtime 1.20.2 -- dependency groupIdorg.apache.flink/groupId artifactIdflink-table-runtime/artifactId version1.20.2/version /dependency dependency groupIdcom.alibaba/groupId artifactIdfastjson/artifactId version1.2.83/version /dependency dependency groupIdorg.slf4j/groupId artifactIdslf4j-api/artifactId version1.7.36/version !-- 使用最新的版本 -- /dependency !-- 根据你的 MySQL 版本选择合适的版本,连接MYSQL必须要引用 -- dependency groupIdmysql/groupId artifactIdmysql-connector-java/artifactId version8.0.33/version /dependency dependency groupIdredis.clients/groupId artifactIdjedis/artifactId version${jedis.version}/version /dependency dependency groupIdjunit/groupId artifactIdjunit/artifactId version3.8.1/version scopetest/scope /dependency /dependencies build plugins !-- Java Compiler -- plugin groupIdorg.apache.maven.plugins/groupId artifactIdmaven-compiler-plugin/artifactId version3.1/version configuration source${target.java.version}/source target${target.java.version}/target /configuration /plugin !-- We use the maven-shade plugin to create a fat jar that contains all necessary dependencies. -- !-- Change the value of mainClass.../mainClass if your program entry point changes. -- plugin groupIdorg.apache.maven.plugins/groupId artifactIdmaven-shade-plugin/artifactId version3.1.1/version executions !-- Run shade goal on package phase -- execution phasepackage/phase goals goalshade/goal /goals configuration createDependencyReducedPomfalse/createDependencyReducedPom artifactSet excludes excludeorg.apache.flink:flink-shaded-force-shading/exclude excludecom.google.code.findbugs:jsr305/exclude excludeorg.slf4j:*/exclude excludeorg.apache.logging.log4j:*/exclude /excludes /artifactSet filters filter !-- Do not copy the signatures in the META-INF folder. Otherwise, this might cause SecurityExceptions when using the JAR. -- artifact*:*/artifact excludes excludeMETA-INF/*.SF/exclude excludeMETA-INF/*.DSA/exclude excludeMETA-INF/*.RSA/exclude /excludes /filter /filters transformers transformer implementationorg.apache.maven.plugins.shade.resource.ServicesResourceTransformer/ transformer implementationorg.apache.maven.plugins.shade.resource.ManifestResourceTransformer mainClassbi.DataStreamJob/mainClass /transformer /transformers /configuration /execution /executions /plugin /plugins pluginManagement plugins !-- This improves the out-of-the-box experience in Eclipse by resolving some warnings. -- plugin groupIdorg.eclipse.m2e/groupId artifactIdlifecycle-mapping/artifactId version1.0.0/version configuration lifecycleMappingMetadata pluginExecutions pluginExecution pluginExecutionFilter groupIdorg.apache.maven.plugins/groupId artifactIdmaven-shade-plugin/artifactId versionRange[3.1.1,)/versionRange goals goalshade/goal /goals /pluginExecutionFilter action ignore/ /action /pluginExecution pluginExecution pluginExecutionFilter groupIdorg.apache.maven.plugins/groupId artifactIdmaven-compiler-plugin/artifactId versionRange[3.1,)/versionRange goals goaltestCompile/goal goalcompile/goal /goals /pluginExecutionFilter action ignore/ /action /pluginExecution /pluginExecutions /lifecycleMappingMetadata /configuration /plugin /plugins /pluginManagement /build /project3 在IDEA本机调试进行下面的设定3.1 在菜单中Run-Edit Configuarations-Modify options解决提示错误找不到steam.api相关的类错误信息如下这是由于quickstart模板中Flink依赖的scope默认为provided, 因此IDEA不会加载这些依赖。dependency groupIdorg.apache.flink/groupId artifactIdflink-streaming-java_${scala.binary.version}/artifactId version${flink.version}/version scopeprovided/scope /dependencyCaused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.api.datastream.DataStream3.2 选择启动的主类在本地调度时提示找不到相关的类文件即出现莫名其妙的错误信息4. flink程序中如果输出内容最好是使用system.out.println在resource输出配置内容使用system.out.println在任何时候都可以显示如果SLF4J 时可能不会显示输出的内容info的内容不会输出private static final Logger logger LoggerFactory.getLogger(MachineRealTimeState.class); logger.info(machine: machineId rungingCount: runningCount 稼动率: availabilityRate);5. 报错信息如下解决方案同第3点解决的问题是一致的。Exception in thread main java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava30/com/google/common/collect/ImmutableList at org.apache.flink.runtime.util.config.memory.ManagedMemoryUtils.clinit(ManagedMemoryUtils.java:57) at org.apache.flink.streaming.api.graph.SimpleTransformationTranslator.configure(SimpleTransformationTranslator.java:121) at org.apache.flink.streaming.api.graph.SimpleTransformationTranslator.translateForStreaming(SimpleTransformationTranslator.java:63) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.translate(StreamGraphGenerator.java:812) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:534) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.getParentInputIds(StreamGraphGenerator.java:833) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.translate(StreamGraphGenerator.java:791) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:534) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.generate(StreamGraphGenerator.java:284) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2523) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2514) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2500) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2319) at org.example.InventorySumBroadcast3.main(InventorySumBroadcast3.java:154) at org.example.App.main(App.java:32) Caused by: java.lang.ClassNotFoundException: org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ... 15 more序号项目教程网址备注1处理启动时找不到类相关的错误IDEA无法运行或调试Flink的解决方式 - 我是小马 - 博客园pom引用范围引的错误
原理与Windows实践)
