Package com.atlan.util

Class AssetBatch

java.lang.Object
com.atlan.util.AssetBatch
All Implemented Interfaces:
Closeable, AutoCloseable
public class AssetBatch extends Object implements Closeable
Utility class for managing bulk updates in batches.

  • Constructor Details

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize, boolean replaceAtlanTags, AssetBatch.CustomMetadataHandling customMetadataHandling)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)
      replaceAtlanTags - if true, all Atlan tags on an existing asset will be overwritten; if false, all Atlan tags will be ignored
      customMetadataHandling - how to handle custom metadata (ignore it, replace it (wiping out anything pre-existing), or merge it)

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize, boolean replaceAtlanTags, AssetBatch.CustomMetadataHandling customMetadataHandling, boolean captureFailures)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)
      replaceAtlanTags - if true, all Atlan tags on an existing asset will be overwritten; if false, all Atlan tags will be ignored
      customMetadataHandling - how to handle custom metadata (ignore it, replace it (wiping out anything pre-existing), or merge it)
      captureFailures - when true, any failed batches will be captured and retained rather than exceptions being raised (for large amounts of processing this could cause memory issues!)

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize, boolean replaceAtlanTags, AssetBatch.CustomMetadataHandling customMetadataHandling, boolean captureFailures, boolean updateOnly)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)
      replaceAtlanTags - if true, all Atlan tags on an existing asset will be overwritten; if false, all Atlan tags will be ignored
      customMetadataHandling - how to handle custom metadata (ignore it, replace it (wiping out anything pre-existing), or merge it)
      captureFailures - when true, any failed batches will be captured and retained rather than exceptions being raised (for large amounts of processing this could cause memory issues!)
      updateOnly - when true, only attempt to update existing assets and do not create any assets (note: this will incur a performance penalty)

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize, boolean replaceAtlanTags, AssetBatch.CustomMetadataHandling customMetadataHandling, boolean captureFailures, boolean updateOnly, boolean track)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)
      replaceAtlanTags - if true, all Atlan tags on an existing asset will be overwritten; if false, all Atlan tags will be ignored
      customMetadataHandling - how to handle custom metadata (ignore it, replace it (wiping out anything pre-existing), or merge it)
      captureFailures - when true, any failed batches will be captured and retained rather than exceptions being raised (for large amounts of processing this could cause memory issues!)
      updateOnly - when true, only attempt to update existing assets and do not create any assets (note: this will incur a performance penalty)
      track - when false, details about each created and updated asset will no longer be tracked (only an overall count of each) -- useful if you intend to send close to (or more than) 1 million assets through a batch

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize, boolean replaceAtlanTags, AssetBatch.CustomMetadataHandling customMetadataHandling, boolean captureFailures, boolean updateOnly, boolean track, boolean caseInsensitive)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)
      replaceAtlanTags - if true, all Atlan tags on an existing asset will be overwritten; if false, all Atlan tags will be ignored
      customMetadataHandling - how to handle custom metadata (ignore it, replace it (wiping out anything pre-existing), or merge it)
      captureFailures - when true, any failed batches will be captured and retained rather than exceptions being raised (for large amounts of processing this could cause memory issues!)
      updateOnly - when true, only attempt to update existing assets and do not create any assets (note: this will incur a performance penalty)
      track - when false, details about each created and updated asset will no longer be tracked (only an overall count of each) -- useful if you intend to send close to (or more than) 1 million assets through a batch
      caseInsensitive - (only applies when updateOnly is true) when matching assets, search for their qualifiedName in a case-insensitive way

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize, boolean replaceAtlanTags, AssetBatch.CustomMetadataHandling customMetadataHandling, boolean captureFailures, boolean updateOnly, boolean track, boolean caseInsensitive, AssetCreationHandling creationHandling)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)
      replaceAtlanTags - if true, all Atlan tags on an existing asset will be overwritten; if false, all Atlan tags will be ignored
      customMetadataHandling - how to handle custom metadata (ignore it, replace it (wiping out anything pre-existing), or merge it)
      captureFailures - when true, any failed batches will be captured and retained rather than exceptions being raised (for large amounts of processing this could cause memory issues!)
      updateOnly - when true, only attempt to update existing assets and do not create any assets (note: this will incur a performance penalty)
      track - when false, details about each created and updated asset will no longer be tracked (only an overall count of each) -- useful if you intend to send close to (or more than) 1 million assets through a batch
      caseInsensitive - (only applies when updateOnly is true) when matching assets, search for their qualifiedName in a case-insensitive way
      creationHandling - if assets are to be created, how they should be created (as full assets or only partial assets)

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize, boolean replaceAtlanTags, AssetBatch.CustomMetadataHandling customMetadataHandling, boolean captureFailures, boolean updateOnly, boolean track, boolean caseInsensitive, AssetCreationHandling creationHandling, boolean tableViewAgnostic)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)
      replaceAtlanTags - if true, all Atlan tags on an existing asset will be overwritten; if false, all Atlan tags will be ignored
      customMetadataHandling - how to handle custom metadata (ignore it, replace it (wiping out anything pre-existing), or merge it)
      captureFailures - when true, any failed batches will be captured and retained rather than exceptions being raised (for large amounts of processing this could cause memory issues!)
      updateOnly - when true, only attempt to update existing assets and do not create any assets (note: this will incur a performance penalty)
      track - when false, details about each created and updated asset will no longer be tracked (only an overall count of each) -- useful if you intend to send close to (or more than) 1 million assets through a batch
      caseInsensitive - (only applies when updateOnly is true) when matching assets, search for their qualifiedName in a case-insensitive way
      creationHandling - if assets are to be created, how they should be created (as full assets or only partial assets)
      tableViewAgnostic - if true, tables and views will be treated interchangeably (an asset in the batch marked as a table will attempt to match a view if not found as a table, and vice versa)

    • AssetBatch

      public AssetBatch(AtlanClient client, int maxSize, boolean replaceAtlanTags, AssetBatch.CustomMetadataHandling customMetadataHandling, boolean captureFailures, boolean updateOnly, boolean track, boolean caseInsensitive, AssetCreationHandling creationHandling, boolean tableViewAgnostic, OffHeapAssetCache created, OffHeapAssetCache updated, OffHeapAssetCache restored, OffHeapAssetCache skipped)
      Create a new batch of assets to be bulk-saved.
      Parameters:
      client - connectivity to Atlan
      maxSize - maximum size of each batch that should be processed (per API call)
      replaceAtlanTags - if true, all Atlan tags on an existing asset will be overwritten; if false, all Atlan tags will be ignored
      customMetadataHandling - how to handle custom metadata (ignore it, replace it (wiping out anything pre-existing), or merge it)
      captureFailures - when true, any failed batches will be captured and retained rather than exceptions being raised (for large amounts of processing this could cause memory issues!)
      updateOnly - when true, only attempt to update existing assets and do not create any assets (note: this will incur a performance penalty)
      track - when false, details about each created and updated asset will no longer be tracked (only an overall count of each) -- useful if you intend to send close to (or more than) 1 million assets through a batch
      caseInsensitive - (only applies when updateOnly is true) when matching assets, search for their qualifiedName in a case-insensitive way
      creationHandling - if assets are to be created, how they should be created (as full assets or only partial assets)
      tableViewAgnostic - if true, tables and views will be treated interchangeably (an asset in the batch marked as a table will attempt to match a view if not found as a table, and vice versa)
      created - off-heap asset cache tracking assets that have been created
      updated - off-heap asset cache tracking assets that have been updated
      restored - off-heap asset cache tracking assets that have been restored
      skipped - off-heap asset cache tracking assets that have been skipped

  • Method Details

    • add

      public AssetMutationResponse add(Asset single) throws AtlanException
      Add an asset to the batch to be processed.
      Parameters:
      single - the asset to add to a batch
      Returns:
      the assets that were created or updated in this batch, or null if the batch is still queued
      Throws:
      AtlanException - on any problems adding the asset to or processing the batch

    • close

      public void close() throws IOException
      Close the batch by freeing up any resources it has used. Note: this will clear any internal caches of results, so only call this after you have processed those!
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException - on any problems freeing up resources

    • flush

      public AssetMutationResponse flush() throws AtlanException
      Flush any remaining assets in the batch.
      Returns:
      the mutation response from the queued batch of assets that were flushed
      Throws:
      AtlanException - on any problems flushing (submitting) the batch

    • getNumCreated

      public AtomicLong getNumCreated()
      Number of assets that were created (no details, just a count).

    • getNumUpdated

      public AtomicLong getNumUpdated()
      Number of assets that were updated (no details, just a count).

    • getNumRestored

      public AtomicLong getNumRestored()
      Number of assets that were potentially restored from being archived, or otherwise touched without actually being updated (no details, just a count).

    • getNumSkipped

      public AtomicLong getNumSkipped()
      Number of assets that were skipped during processing (no details, just a count).

    • getCreated

      public OffHeapAssetCache getCreated()
      Assets that were created (minimal info only).

    • getUpdated

      public OffHeapAssetCache getUpdated()
      Assets that were updated (minimal info only).

    • getRestored

      public OffHeapAssetCache getRestored()
      Assets that were potentially restored from being archived, or otherwise touched without actually being updated (minimal info only).

    • getFailures

      public List<AssetBatch.FailedBatch> getFailures()
      Batches that failed to be committed (only populated when captureFailures is set to true).

    • getSkipped

      public OffHeapAssetCache getSkipped()
      Assets that were skipped, when updateOnly is requested and the asset does not exist in Atlan.

    • getResolvedGuids

      public Map<String,String> getResolvedGuids()
      Map from placeholder GUID to resolved (actual) GUID, for all assets that were processed through the batch.

    • getResolvedQualifiedNames

      public Map<AssetBatch.AssetIdentity,String> getResolvedQualifiedNames()
      Map from all-lowercase qualifiedName (case-insensitive) to case-sensitive qualifiedName, for all assets that were processed through the batch. Note: This is only produced when caseInsensitive is true, otherwise it will be empty.