Product User Community

Asset Ingestion

January 01, 2020
Deltek, Inc
Deltek

Overview

Before reviews, annotations, and approvals can take place, you'll first need to get your assets into ConceptShare. Depending on the nature of your integration and your clients' needs, getting assets into ConceptShare can be accomplished in one of two primary methods. Each method has its own benefits and drawbacks, and your chances of a successful integration increase substantially by understanding the differences and nuances.

Direct Upload

You can push files directly from the host application's storage to ConceptShare as a byte array using the Add*Asset API calls.

  • AddProjectAsset
  • AddVersionedAsset

This is the method to use when your host application can't expose an asset's URL in storage to ConceptShare (ex. some IT security policy or infrastructure limitation that doesn't allow asset URLs to be exposed for ConceptShare). In these cases, you can utilize this method to push the entire file as a byte-arrary in the HTTP Request body to Conceptshare.


CAUTION: The primary limitation of using this method of asset ingestion is that HTTP servers have maximum file size limits for uploads. In our case, that limit is set at 2GB. Using direct HTTP upload, the file size is capped.

This method may be slightly quicker to deploy in the sense that you don't have to worry about security considerations to allow ConceptShare access to the asset store, without also introducing vulnerabilities or unauthorized access. For example, white-listing ConceptShare queue IPs, generating one-time-use URIs, or giving your access token predefined lifespans. This way, the specific asset is exposed for a minimal duration for the queue to download it without exposing the asset to other unknown end-points.

In exchange for not having to secure ConceptShare access to your asset storage, you're inhering the complexity to handle retry logic when uploads fail (since you're not leveraging ConceptShare's queues for downloads), and you're also stuck with a 2 GB payload limit on all uploads.

ConceptShare Direct Asset Upload Sequence Diagram

SOAP API asset ingestion sequence via Direct Upload
NOTE: API calls (being synchronous) will return an asset place holder ID, and you cannot assume successful asset ingestion (nor act on the asset), until you explicitly receive an ASSET_CREATED or ASSET_VERSIONED callback. If some transient network outage impedes your ability to receive any of our attempts to call back your specified endpoint, you should have a watchdog timer that will query the API and checks the status code in order to verify whether the asset is still processing, completed successfully, or failed for some reason.

Read the Event Callbacks guide for more information on event callbacks and their usage. The Callback Reference Documentation has a full listing of all the Callbacks and their respective parameters.

Cloud Storage

If your solution includes cloud storage, such as Amazon's S3, Azure Storage, or any other storage provider that can provide a URL to the resource; you can push your assets to ConceptShare by simply providing that external URI using any of the Add*AssetFromExternallUri calls. This call triggers ConceptShare’s queue service to start a worker thread that will download the file in the background.

  • AddProjectAssetFromExternalUri
  • AddVersionedAssetFromExternalUri

Simply indicate the files that need to be added to ConceptShare, and have the host application provide the URL(s) for ConceptShare to grab those asset(s) from asset storage in the background. Once ConceptShare is finished downloading (and processing the file), it will notify the host application via ASSET_CREATED or ASSET_VERSIONED callbacks, which can then trigger additional notifications to end-users that can now take action.

NOTE: External URI calls returns immediately with an placeholder asset ID and will raise an ASSET_ERROR callback in case of errors, or an ASSET_CREATED/ASSET_VERSIONED callback upon successful completion. In the event that an asset is double versioned (you attempt to version the same file concurrently) your second call will immediately fail with INVALID_PARAMETER.

NOTE: The use of External URI function calls circumvents the upload file size limit of direct uploads. This is because the file is being downloaded from cloud storage, not uploaded via the HTTP Request body.

CAUTION: The external URI needs to be accessible to CS servers. If the asset is password protected, ConceptShare will not be able to access it. This creates the situation wherein assets might be publically accessible, which creates a security and authorization concern. Some mitigation strategies around this problem involve whitelisting CS's server IPs to allow them access, generating one-time-use URIs, setting expiration times on URIs.

WARNING: It's strongly advised that you expire the assets’ access tokens once a successful upload has been confirmed via an ASSET_CREATED or ASSET_VERSIONED callback. In this manner, the asset is not exposed to the open network any longer than it needs to be.

ConceptShare Exteranal URI Asset Upload Sequence Diagram
SOAP API asset ingestion sequence via External URIs
NOTE: API calls (being synchronous) will return an asset place holder ID, and you cannot assume successful asset ingestion (nor act on the asset), until you explicitly receive an ASSET_CREATED or ASSET_VERSIONED callback. If some transient network outage impedes your ability to receive any of our attempts to call back your specified endpoint, you should have a watchdog timer that will query the API and checks the status code in order to verify whether the asset is still processing, completed successfully, or failed for some reason.

For an overview of event callbacks, checkout the guide, and the callback reference documentation has a page listing all the Callbacks and their respective parameters.

Key Components of Asset Ingestion

As the sequence diagrams above will indicate, regardless of whether you use external URIs or push assets directly as byte arrays, your host application will need a mechanism in order to remain informed of the upload state of assets both during and after ingestion. The sequence diagrams for both methods would be adequate if there was never a risk of network disruptions, which is not the case in the real world.

As such, for a comprehensive implementation of asset ingestion you need to account for three things:

  1. A mechanism to initiate asset ingestion in ConceptShare, and retain the asset's place-holder ID
  2. An event handler at an endpoint of your choosing which can receive callback events when the asset has finished processing and is ready for use
  3. Some mechanism to restore synchronization in the event of a failure whereby callbacks cannot reach the host system

Only with these three mechanisms working together can you ensure that the host application and ConceptShare can remain synchronized, even through temporary communication outages or other such transient infrastructure failures.

ConceptShare Integration Asset Ingestion Components
The diagram above illustrates the three key components at work:

  1. Initiating Asset Ingestion


    This component makes API calls to initiate asset uploads. It also retains and stores the asset place-holder ID that's immediately returned by the API. Ideally, it should also track whether the request was to version or upload a net new asset, as well as the time the request was made under a "last checked" column (or something of equivalence). This information should all be tracked in a persistent "pending assets list" for later reference.

  2. Handling Callbacks


    The callback component is just an event handler that can parse and handle ConceptShare callbacks associated with asset ingestion (and optionally others as well). When ConceptShare is finished queuing, uploading, processing, generating proxy formats, creating thumbnails, and all other prerequisite tasks required to display the asset to users, it will trigger the ASSET_CREATED or ASSET_VERSIONED callback. When the callback is received by this component, it should remove the associated entry for that asset in the pending asset list, as all necessary ingestion work is now complete. Whatever other workflow needs to be triggered when assets are ready, can now begin as well. If you're using external URIs, this would be a good time to expire/revoke any access tokens associated with the asset that just completed processing.

  3. Scanning for Stragglers


    This component reasserts synchronization for any assets that are deemed to be in an "unknown" or "dangling" state. By looking at entries in the pending assets list, you can see if any assets ingestion requests were initiated longer than some threshold of your choosing. If you check the time-stamp in the "last checked" column and determine that it's been too long since you heard back, you may consider it a dangling asset. At this point, it's unknown whether the host application missed a callback from ConceptShare, or whether the asset is just very large, or network congestion is just particularly bad that day, and the asset is still processing. Rather than continuing to wait, you can specifically inquire about the asset using the GetAssetProfile API call.

If the asset has uploaded but the callbacks failed to reach the host application, you'll see the asset come back with an Upload Status of None.

If the asset is still processing in some fashion, that too will be reflected in the Upload Status, in which case, you can update the "last checked" field in your pending assets list and you'll either get the callback when things finish or you'll check back when your threshold is reached once more.

If there was an error and the ASSET_ERROR callback did not reach the host application, you'll receive an API error when you use the placeholder ID. This is because ConceptShare will have discarded the asset placeholder ID when ingestion failed.

Ideally, this component would run on a predetermined schedule, either as a scheduled event for each entry added to the pending assets list, or as a maintenance routine which runs at a set interval and checks all remaining entries.


Additional Notes: Networked Storage (Legacy)


CAUTION: The internal URI function calls outlined below are for information purposes only. These API calls cannot be used in integrations using our managed services.

You may have noticed a third set of API functions in the API reference documentation using Local URIs. These are legacy API calls; a throw-back to a time when on-premise deployments of ConceptShare were supported. In those cases, if the deployment included networked storage (Storage Area Network), you could push assets into ConceptShare by their local URI using any of the Add*AssetFromLocalUri calls. This call triggered the background queue service to start a worker thread that would download the file in the background, allowing the user to proceed with other tasks until the upload is completed.

  • AddProjectAssetFromLocalUri
  • AddVersionedAssetFromLocalUri
Deltek, Inc
Deltek