Package com.atlan.model.assets

Class SparkJob

java.lang.Object
com.atlan.model.core.AtlanObject
com.atlan.model.relations.Reference
com.atlan.model.assets.Asset
com.atlan.model.assets.SparkJob
All Implemented Interfaces:
IAsset, ICatalog, IReferenceable, ISpark, ISparkJob, AuditDetail, Serializable, Comparable<Reference>
@Generated("com.atlan.generators.ModelGeneratorV2") public class SparkJob extends Asset implements ISparkJob, ISpark, ICatalog, IAsset, IReferenceable
Instance of a Spark Job run in Atlan.
See Also:

  • Field Details

  • Constructor Details

  • Method Details

    • trimToReference

      public SparkJob trimToReference() throws InvalidRequestException
      Builds the minimal object necessary to create a relationship to a SparkJob, from a potentially more-complete SparkJob object.
      Overrides:
      trimToReference in class Asset
      Returns:
      the minimal object necessary to relate to the SparkJob
      Throws:
      InvalidRequestException - if any of the minimal set of required properties for a SparkJob relationship are not found in the initial object

    • select

      public static FluentSearch.FluentSearchBuilder<?,?> select(AtlanClient client)
      Start a fluent search that will return all SparkJob assets. Additional conditions can be chained onto the returned search before any asset retrieval is attempted, ensuring all conditions are pushed-down for optimal retrieval. Only active (non-archived) SparkJob assets will be included.
      Parameters:
      client - connectivity to the Atlan tenant from which to retrieve the assets
      Returns:
      a fluent search that includes all SparkJob assets

    • select

      public static FluentSearch.FluentSearchBuilder<?,?> select(AtlanClient client, boolean includeArchived)
      Start a fluent search that will return all SparkJob assets. Additional conditions can be chained onto the returned search before any asset retrieval is attempted, ensuring all conditions are pushed-down for optimal retrieval.
      Parameters:
      client - connectivity to the Atlan tenant from which to retrieve the assets
      includeArchived - when true, archived (soft-deleted) SparkJobs will be included
      Returns:
      a fluent search that includes all SparkJob assets

    • refByGuid

      public static SparkJob refByGuid(String guid)
      Reference to a SparkJob by GUID. Use this to create a relationship to this SparkJob, where the relationship should be replaced.
      Parameters:
      guid - the GUID of the SparkJob to reference
      Returns:
      reference to a SparkJob that can be used for defining a relationship to a SparkJob

    • refByGuid

      public static SparkJob refByGuid(String guid, Reference.SaveSemantic semantic)
      Reference to a SparkJob by GUID. Use this to create a relationship to this SparkJob, where you want to further control how that relationship should be updated (i.e. replaced, appended, or removed).
      Parameters:
      guid - the GUID of the SparkJob to reference
      semantic - how to save this relationship (replace all with this, append it, or remove it)
      Returns:
      reference to a SparkJob that can be used for defining a relationship to a SparkJob

    • refByQualifiedName

      public static SparkJob refByQualifiedName(String qualifiedName)
      Reference to a SparkJob by qualifiedName. Use this to create a relationship to this SparkJob, where the relationship should be replaced.
      Parameters:
      qualifiedName - the qualifiedName of the SparkJob to reference
      Returns:
      reference to a SparkJob that can be used for defining a relationship to a SparkJob

    • refByQualifiedName

      public static SparkJob refByQualifiedName(String qualifiedName, Reference.SaveSemantic semantic)
      Reference to a SparkJob by qualifiedName. Use this to create a relationship to this SparkJob, where you want to further control how that relationship should be updated (i.e. replaced, appended, or removed).
      Parameters:
      qualifiedName - the qualifiedName of the SparkJob to reference
      semantic - how to save this relationship (replace all with this, append it, or remove it)
      Returns:
      reference to a SparkJob that can be used for defining a relationship to a SparkJob

    • get

      public static SparkJob get(AtlanClient client, String id) throws AtlanException
      Retrieves a SparkJob by one of its identifiers, complete with all of its relationships.
      Parameters:
      client - connectivity to the Atlan tenant from which to retrieve the asset
      id - of the SparkJob to retrieve, either its GUID or its full qualifiedName
      Returns:
      the requested full SparkJob, complete with all of its relationships
      Throws:
      AtlanException - on any error during the API invocation, such as the NotFoundException if the SparkJob does not exist or the provided GUID is not a SparkJob

    • get

      public static SparkJob get(AtlanClient client, String id, boolean includeAllRelationships) throws AtlanException
      Retrieves a SparkJob by one of its identifiers, optionally complete with all of its relationships.
      Parameters:
      client - connectivity to the Atlan tenant from which to retrieve the asset
      id - of the SparkJob to retrieve, either its GUID or its full qualifiedName
      includeAllRelationships - if true, all the asset's relationships will also be retrieved; if false, no relationships will be retrieved
      Returns:
      the requested full SparkJob, optionally complete with all of its relationships
      Throws:
      AtlanException - on any error during the API invocation, such as the NotFoundException if the SparkJob does not exist or the provided GUID is not a SparkJob

    • get

      public static SparkJob get(AtlanClient client, String id, Collection<AtlanField> attributes) throws AtlanException
      Retrieves a SparkJob by one of its identifiers, with only the requested attributes (and relationships).
      Parameters:
      client - connectivity to the Atlan tenant from which to retrieve the asset
      id - of the SparkJob to retrieve, either its GUID or its full qualifiedName
      attributes - to retrieve for the SparkJob, including any relationships
      Returns:
      the requested SparkJob, with only its minimal information and the requested attributes (and relationships)
      Throws:
      AtlanException - on any error during the API invocation, such as the NotFoundException if the SparkJob does not exist or the provided GUID is not a SparkJob

    • get

      public static SparkJob get(AtlanClient client, String id, Collection<AtlanField> attributes, Collection<AtlanField> attributesOnRelated) throws AtlanException
      Retrieves a SparkJob by one of its identifiers, with only the requested attributes (and relationships).
      Parameters:
      client - connectivity to the Atlan tenant from which to retrieve the asset
      id - of the SparkJob to retrieve, either its GUID or its full qualifiedName
      attributes - to retrieve for the SparkJob, including any relationships
      attributesOnRelated - to retrieve on each relationship retrieved for the SparkJob
      Returns:
      the requested SparkJob, with only its minimal information and the requested attributes (and relationships)
      Throws:
      AtlanException - on any error during the API invocation, such as the NotFoundException if the SparkJob does not exist or the provided GUID is not a SparkJob

    • restore

      public static boolean restore(AtlanClient client, String qualifiedName) throws AtlanException
      Restore the archived (soft-deleted) SparkJob to active.
      Parameters:
      client - connectivity to the Atlan tenant on which to restore the asset
      qualifiedName - for the SparkJob
      Returns:
      true if the SparkJob is now active, and false otherwise
      Throws:
      AtlanException - on any API problems

    • updater

      public static SparkJob.SparkJobBuilder<?,?> updater(String qualifiedName, String name)
      Builds the minimal object necessary to update a SparkJob.
      Parameters:
      qualifiedName - of the SparkJob
      name - of the SparkJob
      Returns:
      the minimal request necessary to update the SparkJob, as a builder

    • trimToRequired

      public SparkJob.SparkJobBuilder<?,?> trimToRequired() throws InvalidRequestException
      Builds the minimal object necessary to apply an update to a SparkJob, from a potentially more-complete SparkJob object.
      Overrides:
      trimToRequired in class Asset
      Returns:
      the minimal object necessary to update the SparkJob, as a builder
      Throws:
      InvalidRequestException - if any of the minimal set of required properties for SparkJob are not found in the initial object

    • removeDescription

      public static SparkJob removeDescription(AtlanClient client, String qualifiedName, String name) throws AtlanException
      Remove the system description from a SparkJob.
      Parameters:
      client - connectivity to the Atlan tenant on which to remove the asset's description
      qualifiedName - of the SparkJob
      name - of the SparkJob
      Returns:
      the updated SparkJob, or null if the removal failed
      Throws:
      AtlanException - on any API problems

    • removeUserDescription

      public static SparkJob removeUserDescription(AtlanClient client, String qualifiedName, String name) throws AtlanException
      Remove the user's description from a SparkJob.
      Parameters:
      client - connectivity to the Atlan tenant on which to remove the asset's description
      qualifiedName - of the SparkJob
      name - of the SparkJob
      Returns:
      the updated SparkJob, or null if the removal failed
      Throws:
      AtlanException - on any API problems

    • removeOwners

      public static SparkJob removeOwners(AtlanClient client, String qualifiedName, String name) throws AtlanException
      Remove the owners from a SparkJob.
      Parameters:
      client - connectivity to the Atlan tenant from which to remove the SparkJob's owners
      qualifiedName - of the SparkJob
      name - of the SparkJob
      Returns:
      the updated SparkJob, or null if the removal failed
      Throws:
      AtlanException - on any API problems

    • updateCertificate

      public static SparkJob updateCertificate(AtlanClient client, String qualifiedName, CertificateStatus certificate, String message) throws AtlanException
      Update the certificate on a SparkJob.
      Parameters:
      client - connectivity to the Atlan tenant on which to update the SparkJob's certificate
      qualifiedName - of the SparkJob
      certificate - to use
      message - (optional) message, or null if no message
      Returns:
      the updated SparkJob, or null if the update failed
      Throws:
      AtlanException - on any API problems

    • removeCertificate

      public static SparkJob removeCertificate(AtlanClient client, String qualifiedName, String name) throws AtlanException
      Remove the certificate from a SparkJob.
      Parameters:
      client - connectivity to the Atlan tenant from which to remove the SparkJob's certificate
      qualifiedName - of the SparkJob
      name - of the SparkJob
      Returns:
      the updated SparkJob, or null if the removal failed
      Throws:
      AtlanException - on any API problems

    • updateAnnouncement

      public static SparkJob updateAnnouncement(AtlanClient client, String qualifiedName, AtlanAnnouncementType type, String title, String message) throws AtlanException
      Update the announcement on a SparkJob.
      Parameters:
      client - connectivity to the Atlan tenant on which to update the SparkJob's announcement
      qualifiedName - of the SparkJob
      type - type of announcement to set
      title - (optional) title of the announcement to set (or null for no title)
      message - (optional) message of the announcement to set (or null for no message)
      Returns:
      the result of the update, or null if the update failed
      Throws:
      AtlanException - on any API problems

    • removeAnnouncement

      public static SparkJob removeAnnouncement(AtlanClient client, String qualifiedName, String name) throws AtlanException
      Remove the announcement from a SparkJob.
      Parameters:
      client - connectivity to the Atlan client from which to remove the SparkJob's announcement
      qualifiedName - of the SparkJob
      name - of the SparkJob
      Returns:
      the updated SparkJob, or null if the removal failed
      Throws:
      AtlanException - on any API problems

    • replaceTerms

      public static SparkJob replaceTerms(AtlanClient client, String qualifiedName, String name, List<IGlossaryTerm> terms) throws AtlanException
      Replace the terms linked to the SparkJob.
      Parameters:
      client - connectivity to the Atlan tenant on which to replace the SparkJob's assigned terms
      qualifiedName - for the SparkJob
      name - human-readable name of the SparkJob
      terms - the list of terms to replace on the SparkJob, or null to remove all terms from the SparkJob
      Returns:
      the SparkJob that was updated (note that it will NOT contain details of the replaced terms)
      Throws:
      AtlanException - on any API problems

    • appendTerms

      @Deprecated public static SparkJob appendTerms(AtlanClient client, String qualifiedName, List<IGlossaryTerm> terms) throws AtlanException
      Deprecated.
      see Asset.AssetBuilder.appendAssignedTerm(GlossaryTerm)
      Link additional terms to the SparkJob, without replacing existing terms linked to the SparkJob. Note: this operation must make two API calls — one to retrieve the SparkJob's existing terms, and a second to append the new terms.
      Parameters:
      client - connectivity to the Atlan tenant on which to append terms to the SparkJob
      qualifiedName - for the SparkJob
      terms - the list of terms to append to the SparkJob
      Returns:
      the SparkJob that was updated (note that it will NOT contain details of the appended terms)
      Throws:
      AtlanException - on any API problems

    • removeTerms

      @Deprecated public static SparkJob removeTerms(AtlanClient client, String qualifiedName, List<IGlossaryTerm> terms) throws AtlanException
      Deprecated.
      see Asset.AssetBuilder.removeAssignedTerm(GlossaryTerm)
      Remove terms from a SparkJob, without replacing all existing terms linked to the SparkJob. Note: this operation must make two API calls — one to retrieve the SparkJob's existing terms, and a second to remove the provided terms.
      Parameters:
      client - connectivity to the Atlan tenant from which to remove terms from the SparkJob
      qualifiedName - for the SparkJob
      terms - the list of terms to remove from the SparkJob, which must be referenced by GUID
      Returns:
      the SparkJob that was updated (note that it will NOT contain details of the resulting terms)
      Throws:
      AtlanException - on any API problems

    • appendAtlanTags

      @Deprecated public static SparkJob appendAtlanTags(AtlanClient client, String qualifiedName, List<String> atlanTagNames) throws AtlanException
      Deprecated.
      see Asset.AssetBuilder.appendAtlanTags(List)
      Add Atlan tags to a SparkJob, without replacing existing Atlan tags linked to the SparkJob. Note: this operation must make two API calls — one to retrieve the SparkJob's existing Atlan tags, and a second to append the new Atlan tags.
      Parameters:
      client - connectivity to the Atlan tenant on which to append Atlan tags to the SparkJob
      qualifiedName - of the SparkJob
      atlanTagNames - human-readable names of the Atlan tags to add
      Returns:
      the updated SparkJob
      Throws:
      AtlanException - on any API problems

    • appendAtlanTags

      @Deprecated public static SparkJob appendAtlanTags(AtlanClient client, String qualifiedName, List<String> atlanTagNames, boolean propagate, boolean removePropagationsOnDelete, boolean restrictLineagePropagation) throws AtlanException
      Deprecated.
      see Asset.AssetBuilder.appendAtlanTags(List, boolean, boolean, boolean, boolean)
      Add Atlan tags to a SparkJob, without replacing existing Atlan tags linked to the SparkJob. Note: this operation must make two API calls — one to retrieve the SparkJob's existing Atlan tags, and a second to append the new Atlan tags.
      Parameters:
      client - connectivity to the Atlan tenant on which to append Atlan tags to the SparkJob
      qualifiedName - of the SparkJob
      atlanTagNames - human-readable names of the Atlan tags to add
      propagate - whether to propagate the Atlan tag (true) or not (false)
      removePropagationsOnDelete - whether to remove the propagated Atlan tags when the Atlan tag is removed from this asset (true) or not (false)
      restrictLineagePropagation - whether to avoid propagating through lineage (true) or do propagate through lineage (false)
      Returns:
      the updated SparkJob
      Throws:
      AtlanException - on any API problems

    • removeAtlanTag

      @Deprecated public static void removeAtlanTag(AtlanClient client, String qualifiedName, String atlanTagName) throws AtlanException
      Deprecated.
      see Asset.AssetBuilder.removeAtlanTag(String)
      Remove an Atlan tag from a SparkJob.
      Parameters:
      client - connectivity to the Atlan tenant from which to remove an Atlan tag from a SparkJob
      qualifiedName - of the SparkJob
      atlanTagName - human-readable name of the Atlan tag to remove
      Throws:
      AtlanException - on any API problems, or if the Atlan tag does not exist on the SparkJob

    • _internal

      public static SparkJob.SparkJobBuilder<?,?> _internal()

    • toBuilder

      public SparkJob.SparkJobBuilder<?,?> toBuilder()
      Overrides:
      toBuilder in class Reference

    • getInputToAirflowTasks

      public SortedSet<IAirflowTask> getInputToAirflowTasks()
      Tasks to which this asset provides input.
      Specified by:
      getInputToAirflowTasks in interface ICatalog
      Specified by:
      getInputToAirflowTasks in interface ISpark
      Specified by:
      getInputToAirflowTasks in interface ISparkJob

    • getInputToProcesses

      public SortedSet<ILineageProcess> getInputToProcesses()
      Processes to which this asset provides input.
      Specified by:
      getInputToProcesses in interface ICatalog
      Specified by:
      getInputToProcesses in interface ISpark
      Specified by:
      getInputToProcesses in interface ISparkJob

    • getInputToSparkJobs

      public SortedSet<ISparkJob> getInputToSparkJobs()
      TBC
      Specified by:
      getInputToSparkJobs in interface ICatalog
      Specified by:
      getInputToSparkJobs in interface ISpark
      Specified by:
      getInputToSparkJobs in interface ISparkJob

    • getInputs

      public SortedSet<ICatalog> getInputs()
      Assets that are inputs to this task.
      Specified by:
      getInputs in interface ISparkJob

    • getModelImplementedAttributes

      public SortedSet<IModelAttribute> getModelImplementedAttributes()
      Attributes implemented by this asset.
      Specified by:
      getModelImplementedAttributes in interface ICatalog
      Specified by:
      getModelImplementedAttributes in interface ISpark
      Specified by:
      getModelImplementedAttributes in interface ISparkJob

    • getModelImplementedEntities

      public SortedSet<IModelEntity> getModelImplementedEntities()
      Entities implemented by this asset.
      Specified by:
      getModelImplementedEntities in interface ICatalog
      Specified by:
      getModelImplementedEntities in interface ISpark
      Specified by:
      getModelImplementedEntities in interface ISparkJob

    • getOutputFromAirflowTasks

      public SortedSet<IAirflowTask> getOutputFromAirflowTasks()
      Tasks from which this asset is output.
      Specified by:
      getOutputFromAirflowTasks in interface ICatalog
      Specified by:
      getOutputFromAirflowTasks in interface ISpark
      Specified by:
      getOutputFromAirflowTasks in interface ISparkJob

    • getOutputFromProcesses

      public SortedSet<ILineageProcess> getOutputFromProcesses()
      Processes from which this asset is produced as output.
      Specified by:
      getOutputFromProcesses in interface ICatalog
      Specified by:
      getOutputFromProcesses in interface ISpark
      Specified by:
      getOutputFromProcesses in interface ISparkJob

    • getOutputFromSparkJobs

      public SortedSet<ISparkJob> getOutputFromSparkJobs()
      TBC
      Specified by:
      getOutputFromSparkJobs in interface ICatalog
      Specified by:
      getOutputFromSparkJobs in interface ISpark
      Specified by:
      getOutputFromSparkJobs in interface ISparkJob

    • getOutputs

      public SortedSet<ICatalog> getOutputs()
      Assets that are outputs from this task.
      Specified by:
      getOutputs in interface ISparkJob

    • getProcess

      public ILineageProcess getProcess()
      TBC
      Specified by:
      getProcess in interface ISparkJob

    • getSparkAppName

      public String getSparkAppName()
      Name of the Spark app containing this Spark Job For eg. extract_raw_data
      Specified by:
      getSparkAppName in interface ISparkJob

    • getSparkMaster

      public String getSparkMaster()
      The Spark master URL eg. local, local[4], or spark://master:7077
      Specified by:
      getSparkMaster in interface ISparkJob

    • getSparkOrchestratedByAirflowAssets

      public SortedSet<IAirflow> getSparkOrchestratedByAirflowAssets()
      Airflow assets that execute this spark asset.
      Specified by:
      getSparkOrchestratedByAirflowAssets in interface ISpark
      Specified by:
      getSparkOrchestratedByAirflowAssets in interface ISparkJob

    • getSparkRunEndTime

      public Long getSparkRunEndTime()
      End time of the Spark Job eg. 1695673598218
      Specified by:
      getSparkRunEndTime in interface ISpark
      Specified by:
      getSparkRunEndTime in interface ISparkJob

    • getSparkRunOpenLineageState

      public OpenLineageRunState getSparkRunOpenLineageState()
      OpenLineage state of the Spark Job run eg. COMPLETE
      Specified by:
      getSparkRunOpenLineageState in interface ISpark
      Specified by:
      getSparkRunOpenLineageState in interface ISparkJob

    • getSparkRunOpenLineageVersion

      public String getSparkRunOpenLineageVersion()
      OpenLineage Version of the Spark Job run eg. 1.1.0
      Specified by:
      getSparkRunOpenLineageVersion in interface ISpark
      Specified by:
      getSparkRunOpenLineageVersion in interface ISparkJob

    • getSparkRunStartTime

      public Long getSparkRunStartTime()
      Start time of the Spark Job eg. 1695673598218
      Specified by:
      getSparkRunStartTime in interface ISpark
      Specified by:
      getSparkRunStartTime in interface ISparkJob

    • getSparkRunVersion

      public String getSparkRunVersion()
      Spark Version for the Spark Job run eg. 3.4.1
      Specified by:
      getSparkRunVersion in interface ISpark
      Specified by:
      getSparkRunVersion in interface ISparkJob

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Asset

    • canEqual

      protected boolean canEqual(Object other)
      Overrides:
      canEqual in class Asset

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Asset

    • toString

      public String toString()
      Description copied from class: AtlanObject
      Overrides:
      toString in class Asset

    • getTypeName

      public String getTypeName()
      Fixed typeName for SparkJobs.
      Specified by:
      getTypeName in interface AuditDetail
      Specified by:
      getTypeName in interface IAsset
      Specified by:
      getTypeName in interface ICatalog
      Specified by:
      getTypeName in interface IReferenceable
      Specified by:
      getTypeName in interface ISpark
      Specified by:
      getTypeName in interface ISparkJob
      Overrides:
      getTypeName in class Reference