job中执行每行
job中执行每行 有时,我们需要job或转换执行多次,且每次传入的参数都不同。假如你正在做数据迁移的工作,需要导入每天的旧数据,则需要一个job根据指定的日期导入数据,该日期被指定作为参数,如果我们需要导入从一个起始日期到昨天所有的旧数据,手工运行j
job中执行每行
有时,我们需要job或转换执行多次,且每次传入的参数都不同。假如你正在做数据迁移的工作,需要导入每天的旧数据,则需要一个job根据指定的日期导入数据,该日期被指定作为参数,如果我们需要导入从一个起始日期到昨天所有的旧数据,手工运行job是痛苦的也是错误的做法,kettle是可以首先计算正确的日期然后根据每个日期运行导入job。
在主job中,首先运行返回日期转换(每个日期作为一行),实际导入数据的job放在转换的后面执行,并且根据转换的结果执行每一行,job可以被设置从一个结果字段中获取参数。
生成日期
这个转换负责生成只有一个日期字段的数据行,“set start date ”步骤生成10000行数据流(根据实际情况估算行数,测试时可以设置小点),包括一个日期类型常量字段。“calc days”步骤插入一个序列到数据流中,为了在“calc date”步骤可以计算连续的日期;其后的步骤去除掉其他的辅助字段,然后增加昨天的日期到数据流中;接着是一个比较功能的步骤,判断每行的中日期是否早于今天,如果进入“discard”步骤,否则进入“copy rows to result”步骤。
主job
主job中执行生成日期转换之后,是子job,负责执行每个输入日期,该job必须要明确设置去执行每个输入行,即设置“execute for every input row”复选框选中。同时需要传入一个计算日期,通过“date”参数,所以“copy previous results to parameters?”也需要选中。最后参数必须从输入行中映射,参数名和输入字段都命名为“date”。
子job
子job接收每行的“date”日期参数并运行,示例中简单使用date参数记录日志然后完成。
下载
这里下载本示例,主job运行子job,针对从2010-01-01到昨天的所有行。需要指出的是,本示例的技术并不适合所有的场合,特别“循环直到条件为真”(不确定次数),这时可能用到这篇文章的方法去实现循环。

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics



Automation and task scheduling play a vital role in streamlining repetitive tasks in software development. Imagine there is a Python script that needs to be executed every 5 minutes, such as getting data from an API, performing data processing, or sending periodic updates. Running scripts manually so frequently can be time-consuming and error-prone. This is where task scheduling comes in. In this blog post, we will explore how to schedule a Python script to execute every 5 minutes, ensuring it runs automatically without manual intervention. We will discuss different methods and libraries that can be used to achieve this goal, allowing you to automate tasks efficiently. An easy way to run a Python script every 5 minutes using the time.sleep() function is to utilize tim

How to use Python to write and execute scripts in Linux In the Linux operating system, we can use Python to write and execute various scripts. Python is a concise and powerful programming language that provides a wealth of libraries and tools to make scripting easier and more efficient. Below we will introduce the basic steps of how to use Python for script writing and execution in Linux, and provide some specific code examples to help you better understand and use it. Install Python

XXL-JOB Description XXL-JOB is a lightweight distributed task scheduling platform. Its core design goals are rapid development, easy learning, lightweight, and easy expansion. The source code is now open and connected to the online product lines of many companies, ready to use out of the box. 1. Vulnerability details The core issue of this vulnerability is GLUE mode. XXL-JOB supports multi-language and script tasks through "GLUE mode". The task features of this mode are as follows: ●Multi-language support: supports Java, Shell, Python, NodeJS, PHP, PowerShell... and other types. ●WebIDE: Tasks are maintained in the dispatch center in source code mode and support online development and maintenance through WebIDE. ●Dynamic effective: user online communication

What configurations are needed to use CAD smoothly? To use CAD software smoothly, you need to meet the following configuration requirements: Processor requirements: In order to run "Word Play Flowers" smoothly, you need to be equipped with at least one Intel Corei5 or AMD Ryzen5 or above processor. Of course, if you choose a higher-performance processor, you'll be able to get faster processing speeds and better performance. Memory is a very important component in the computer. It has a direct impact on the performance and user experience of the computer. Generally speaking, we recommend at least 8GB of memory, which can meet the needs of most daily use. However, for better performance and smoother usage experience, it is recommended to choose a memory configuration of 16GB or above. This ensures that the

The Brown-Forsythe test is a statistical test used to determine whether the variances of two or more groups are equal. Levene's test uses the absolute deviation from the mean, while the Brown-Forsythe test uses the deviation from the median. The null hypothesis used in the test is as follows - H0: The variances of the groups (population) are equal. The alternative hypothesis is that the variances of the groups (population) are not equal. - H1: The variances of the groups (population) are not equal. To perform the test, we calculate the median of each group and its correlation The absolute deviation of the number of digits. We then calculate the F-statistic based on the variance of these deviations. Assume that the calculated F statistic is greater than the critical value in the F distribution table. In this case, we reject the null hypothesis and conclude that the variances of the groups are not equal. In Python, sc

In this article, we will learn about the lambda function in Python and why we need it, and see some practical examples of lambda functions. What is lambda function in Python? A Lambda function is often called an "anonymous function" and is the same as a normal Python function, except that it can be defined without a name. The >def keyword is used to define ordinary functions, while the lambda keyword is used to define anonymous functions. However, they are limited to single-line expressions. They, like regular functions, can accept multiple arguments. Syntax lambdaarguments:expression This function accepts any number of inputs, but only evaluates and returns an expression. Lamb

Computers are one of the indispensable tools in our daily lives and work, but sometimes we encounter some problems, such as the computer shutting down very slowly and spinning in circles. This situation is very common among many people, but for many people, they don’t know what to do. First, we need to understand the reason why the computer shuts down slowly. There are several possible reasons why this problem may occur. First of all, there may be some running programs or processes in the computer that have not been closed. As a result, when the computer shuts down, it needs to wait for these programs or processes to close before it can shut down successfully.

Using light to train neural networks, Tsinghua University results were recently published in Nature! What should I do if I cannot apply the backpropagation algorithm? They proposed a Fully Forward Mode (FFM) training method that directly performs the training process in the physical optical system, overcoming the limitations of traditional digital computer simulations. To put it simply, it used to be necessary to model the physical system in detail and then simulate these models on a computer to train the network. The FFM method eliminates the modeling process and allows the system to directly use experimental data for learning and optimization. This also means that training no longer needs to check each layer from back to front (backpropagation), but can directly update the parameters of the network from front to back. To use an analogy, like a puzzle, backpropagation
