It’s worth taking the time to understand the parameters of BaseOperator Internal object based on the information persisted in the sensor_instance table.All operators are derived from BaseOperator and acquire muchįunctionality through inheritance. Refresh sensor instances need to be handled by this operator. Ui_color = #e6f1f2 ¶ _validate_input_values ( self ) ¶ _load_sensor_works ( self, session = None ) ¶ Poke_timeout ( float) - Time, in seconds before the task times out and fails.
MAX AIRFLOW SENSOR CODE
Shard_max ( int) - shard code upper bound (exclusive) Shard_min ( int) - shard code lower bound (inclusive) Job times out if poke_timeout is not defined. Smart_sensor_timeout ( float) - Time, in seconds before the internal sensor Poke_interval ( int) - Time in seconds that the job should wait in Soft_fail ( bool) - Set to true to mark the task as SKIPPED on failure When a criteria is met or fail by time out, it updateĪll sensor task state in task_instance table Parameters Smart sensor operators poke for all sensor tasks in the dictionary atĪ time interval. Smart Sensor operators keep refresh a dictionary by visiting DB. Smart sensor operators are derived from this class. SmartSensorOperator ( poke_interval = 180, smart_sensor_timeout = 60 * 60 * 24 * 7, soft_fail = False, shard_min = 0, shard_max = 100000, poke_timeout = 6.0, * args, ** kwargs ) ¶īases:, :return: should_fail_current_run ( self ) ¶ Returnsīoolean is_expired ( self ) ¶ Returns Set the time point when the sensor should be failed if it kept getting infra Retry poke function without failing current task run. There is a retry window _infra_failure_retry_window that the smart sensor will Is_infra_failure - If current exception was caused by transient infra failure. ParametersĮxception_info - Details of the exception information. There is a 6 hours windowįor retry without failing current run. Which was set at the first infra failure exception arrives. Implies an infra failure, this function will check the recorded infra failure timeout This function set the latest exception information for sensor exception. If the exception is an infra failure Typeīoolean set_latest_exception ( self, exception_info, is_infra_failure = False ) ¶ Infra failure, give the task more chance to retry before fail it. Hold sensor exception information and the type of exception.
SensorExceptionInfo ( exception_info, is_infra_failure = False, infra_failure_retry_window = datetime.timedelta(minutes=130) ) ¶ The cached task object expires if there is no poke for 20 minutes. Mark this poke work to be popped from cached dict after current loop. clear_state ( self ) ¶Ĭlear state for cached poke work. To_flush: If we should flush the cached work.
Last_poke_time: The latest time this cached work being called. Sensor_task: The cached object for executing the poke function. The sensor_task used to poke and recent poke result state. Wrapper class for the poke work inside smart sensor. close_sensor_logger ( self ) ¶Ĭlose log handler for a sensor work. Return logger for a sensor instance object. :return: log handler _get_sensor_logger ( self, si ) ¶ _eq_ ( self, other ) ¶ static create_new_task_handler ( ) ¶Ĭreate task log handler for a sensor work. Key used to query in smart sensor for cached sensor work. Key for the task instance that maps to the sensor work. Log: The sensor work logger which will mock the corresponding task instance log. Op_classpath: The sensor operator class pathĮncoded_poke_context: The raw data from sensor_instance poke_context column. Hashcode: This is the signature of poking job.
Poke_context: Decoded poke_context for the sensor task.Įxecution_context: Decoded execution_context. Create a sensor work based on sensor instance record.Ī sensor work object has the following attributes:Įxecution_date: sensor_instance execution_date. This class stores a sensor work with decoded context value.
0 kommentar(er)
