FastClip/Partial File Restore

Net-X-Code Partial File Restore


Net-X-Code's partial file restore provides a complete, index/proxy/metadata/restore capability for Windows, Linux and OS-X.

Net-X-Code Partial file restore overview

Indexing

Net-X-Code uses a real time index system to quickly produce small, pointer files that track the actual media within almost any broadcast or post media file.  This file is small enough to keep on fast, local or network storage, and alleviates the need to look at the source file tables/metadata/extra data.  When the part of the file that needs to be restored is determined, from time code or absolute time code, the index file can be used to read only the parts of the file needed to create the new partial file.

Source Files and Proxies

The restoration system includes support for MXF, GXF, LXF, MOV, AVI, DPX, CINE, RMF, VFW, CDX, and all other files supported by Drastic's software (for a full list, please see file format support in videoQC) Along with indexing, the system can produce proxy files with matching metadata and time code.  The proxy files can be read with standard preview players (QuickTime, Windows Media Player, VLC, etc.) and Drastic players that work on all major platforms and/or run with time code across a custom HTML5 web player.  The time code in the proxy can be used to create in and out points to be used on the main source files.

netxplayer

 

Google/Amazon/Cloud, SMB, NFS, NAS, HTTP, FTP and SAN Support

The offline/nearline source files can be accessed via a number of access methods.  One of the main methods is via the cloud, including FTP and HTTP direct access.  Shared file systems like SMB/CIFS, mapped HTTP and NFS files are also accessed directly.  SAN or attached storage can be any OS available drive.  SGI's CXFS is particularly supported, including automatic access to the DMF tape file and restore API.  If using nearline or cloud based storage, the index files are optional.  They significantly speed up initial open, but restore speed is equally fast with or without them.

HTTP/RESTful API, Single or Multi Server Configs

Restore groups can be contained on one or more servers.  Each daemon process can run independently or as a group on a server.  The central server process exports an HTTP RESTful/AJAX interface that can be used from within a web browser, or be controlled by user API code in any programming language that supports HTTP get and XML.

FastClip

FastClip is a system in PFR and Net-X-Code that allows clipping of files and live streams for direct output to web servers.  This allows extremely fast turnaround of video assets to external servers or even full resolution assets to be created for immediate use in editing or on-air run down.  Because everything is based on time code and index files, different level/bitrate files can be clipped simultaneously to create multi bitrate video for multiple devices and bandwidth devices.

Extra Functions

To make the system more complete, a number of accessory functions are also available:

  • Thumbnail generation
  • Metadata extraction
  • Closed Caption extraction
  • Wave/Audio extraction

Workflow Overview

Net X Copy PartialFileRestoreWorkflow 

The partial file restore begins with a source file.  Almost any broadcast, post production or web based file is supported (for a full list, see the file format list in videoQC).  An RTIndex is created from the original file.  This small pointer file is very quick to create and allows fast direct access to the media in the source file.  This file can be stored on a local drive, NAS or on the tape/cloud.  A secondary process can, at the same time, create a frame for frame matching proxy file with matching metadata and time code.  These files are normally moved to a web facing database server.  The file can be played back via HTML5 video with time code, or with many other players.  The primary file is then moved off to tape, NAS, cloud, AWS or other long term storage.  When a section of a file is needed, the proxy file can be used to determine the in and out time code of the segment.  These time code points are then used to reference the original file.  That section of the file can then be used to create a new file, either with no conversion (re-wrap) or converting it to another standard format.  Only the minimum data required is actually read from the source file.  With SAN systems like CXFS, the required bytes are automatically recalled to make the system even more efficient.

Typical Proxy, Convert and PFR Session

This section has some typical proxy, convert and partial file restore workflows. The first scenario is a complete list. The other scenarios assume the first few steps in the first scenario have already been accomplished.

Scenario 1 – full access

  • The file (MXF, MOV, AVI, CINE, etc) arrives at ingest
  • A “command=copy&profile=index” command is sent to index to original filename
  • A “command=copy&profile=mp4-h264” command is sent to make a proxy file with time code, multitrack audio, closed captions, metadata and proxy index filename
  • Two “command=copy&pisrc&pidst” commands are sent to create JPEG images for the source and proxy files
  • At this point, the proxy and main index can be stored in a real or near real time storage, and the main file may be moved to long term storage, tape, cloud (google/s3) or other offline storage
  • The user used the HTML5 player’s time code (or other timecode source) to set one or more In and Out points on the file that needs to be restored.
  • A “command=copy&profile=wrap” is sent to access the bytes of the original file and create a new file of the same type, without any recompression of audio or video, at the target location<

Scenario 2 – tape restore

  • This assumes the basic processing in Scenario 1 has been done
  • A “command=getcopyinout” is sent with the absolute or time code based in and out points, and the index of source file it will come from
  • This returns a series of one or more file names with start and end byte locations
  • At this point, the controller restores those byte areas of the files to the name specified by the return
  • Once the areas are restored, a “command=copy&profile=wrap” is sent along with the temp folder to create the new output filename

Scenario 3 – cloud restore

  • This assumes the basic processing in Scenario 1 has been done
  • If the index file is stored on cloud, it can be restored locally first, or read directly from the cloud (https, ftps, aws)
  • If the main file is in Glacier, then a command will be sent to restore the section needed to S3 before the restore is done
  • Once there is access to the file, or file part, the “command=copy&profile=wrap” can be called normally
  • If the resource is in available cloud storage (e.g. not Glacier), then partial file restores may be done from the original file without indexing it first. This will cause more data to be read, but only the headers and tables necessary to find the audio/video/data the restoration needs

Scenario 4 – in line conversions

  • This assumes the basic processing in Scenario 1 has been done
  • For any restore scenario, the file restored can be a byte accurate re-wrap of the original into a new container, or it can be translated in process (on the fly) to any supported standard format. These formats include MXF Op1a, Op-Atom, P2, IMX, D11, IMF, MOV, Uncompressed and many other containers, with codecs including JPEG-2000, XDCam, MPEG-2, h.264, HEVC, AVCi 100/200, XAVC-S. XAVC, Long-G, TR-01, DV and many others
  • The commands can also be used with or without index files to convert all our part of local clips to any of these formats

 Partial List of Supported Formats

Please contact us for the most up to date list.

  • MXF - OP1a, OP1b, OP-Atom, EVS, Uncompressed, AVC, XAVC, AVCi, DVHD/25/50, XDCam, MPEG-2, IMX, h.264, HEVC, Sony Raw, HDCam, F5/F55, Long-G, etc.
  • MOV - ProRes, DVHD/25/50, AVCi, h.264, HEVC, most QuickTime codecs
  • MP4 - h.264 (4:2:0, 4:2:2, 8/10/12 bit), HEVC  (4:2:0, 4:2:2, 8/10/12 bit), Sony XDCam
  • AVI - Uncompressed, h.264, MJPEG, Uncompressed, MPEG-2, DVHD/25/50
  • Avid DS Gen - Uncompressed
  • Cine - Raw bayer
  • CRM - Canon raw bayer
  • DVS - Uncompressed RGB or YCbCr
  • GVK2 - Grass Valley K2 file groups
  • GXF - SMPTE 360
  • YUV/RAW - raw uncompressed data
  • IHSS - Speed grade uncompressed
  • TR-01 - JPEG-2000 TS-01/Evertz transport streams
  • KRW - Kinefinity raw bayer
  • LXF - Harris MPEG-2, DVHD/25/50, AVCi-100
  • MLV - Magic Lantern raw bayer
  • TS - MPEG-1, MPEG-2, JPEG-2000, h.264, HEVC transport streams
  • MTS - h.264 camera captures
  • R3D - Red camera files
  • SEQ/SIV - Raw bayer camera files
  • VCAP - Tektronix raw files
  • DNG/KRW/VRW/CDX - Raw bayer sequence of stills
  • DPX/CIN//HDR/SGITIFF/PNG/PSD/TGA/BMP - Image sequences

Net-X-Code API (includes PFR, Net-X-Convert, Net-X-Copy)

Net-X-Copy API (pdf)