fbpx

Net-X-Copy - Partial File Restore

verifycc icon

Net-X-Copy is part of the Net-X-Convert/Net-X-Proxy workstation and Net-X-Code server software. It is designed to convert audio/video files from one type to another and to perform partial file restores from indexed and non indexed files on disk, tape or cloud. This article contains the Linux/OS-X/Windows command line API for this tool.  A RESTful API is also available.

Net-X-Copy - Partial File Restore

Net-X-Copy supports partial file restore for all the standard broadcast and postproduction formats we support without the need for a mezzanine or intermediate file format.  The original files are indexed, a timecode accurate proxy is created, and then any section of the original file can be recalled via timecode or absolute offsets based on HTML5 playback of the proxy.  The high resolution asset can be stored on near line storage, cloud storage (HTTPS/FTP/Amazon/Google/SMB), tape or any other kind of block, object or file storage your server can access.

There are four modes the partial file restore can operate in:

  1. Full online.  This is the simplest mode where access to the source file is fast.  Here a call can be made with absolute offset values or timecode values to determine the area to restore, and the header/metadata/index in the main file will be used to find the audio/video data and produce the newly wrapped/restored file in the same format.
  2. Live online:  For live files like MXF Open, transport streams or other growing files, an area can be clipped or restored while the file is still writing.  This includes local, network and cloud sources.  Here the header/metadata is pulled from the file and the index is calculated based on absolute or timecode values to find the media for the new file.
  3. Slow online:  For files that are stored on very slow storage, like Amazon's Glacier or other cold storage systems, an index for the file should first be generated.  The file is pushed to long term storage, and the small index file is kept locally available.  To restore a clip, the index file is used to calculate the minimum amout of bytes required to create the new file.  This byte range can then be retreived from the file during the restore by Net-X-Copy.
  4. Full offline.  When the file is completely offline, using tapes, stored hard drives or other methods, an index must be generated before the file is stored.  This index can then be used to calculate the minimum amount of the file (as a byte range) that needs to be retrieved to create the restore/wrap file.  Once this byte range is available on temporary local storage, the actual restore is called to make the new file.

The index/proxy/restore can be controlled via our RESTful API in www.net-x-code.com or via the command line parameters descibed below.


Typical Scenarios

This section has some typical proxy, convert and partial file restore workflows. The first scenario is a complete list. The subsequent scenarios assume the first few steps in the first scenario have already been accomplished

Scenario 1 - full access/on line source

  • The file (MXF, MOV, AVI, CINE, etc) arrives at ingest
  • A “command=copy&profile=index” command is sent to index to original filename
  • A “command=copy&profile=mp4-h264” command is sent to make a proxy file with time code, multitrack audio, closed captions, metadata and proxy index filename
  • Two “command=copy&pisrc&pidst” commands are sent to create JPEG images for the source and proxy files
  • At this point, the proxy and main index can be stored in a real or near real time storage, and the main file may be moved to long term storage, tape, cloud (google/s3) or other offline storage
  • The user uses the HTML5 player’s time code (or other time code source) to set one or more In and Out points on the file that needs to be restored
  • A “command=copy&profile=wrap” is sent to access the bytes of the original file and create a new file of the same type, without any recompression of audio or video, at the target location

Scenario 2 - tape restore

  • This assumes the basic processing in Scenario 1 has been done
  • A “command=getcopyinout” is sent with the absolute or time code based in and out points, and the index of source file it will come from
  • This returns a series of one or more file names with start and end byte locations
  • At this point, the controller restores those byte areas of the files to the name specified by the return
  • Once the areas are restored, a “command=copy&profile=wrap” is sent along with the temp folder to create the new output filename

Scenario 3 - cloud restore

  • This assumes the basic processing in Scenario 1 has been done
  • If the index file is stored on cloud, it can be restored locally first, or read directly from the cloud (https, ftps, aws)
  • If the main file is in Glacier, then a command will be sent to restore the section needed to S3 before the restore is done
  • Once there is access to the file, or file part, the “command=copy&profile=wrap” can be called normally
  • If the resource is in available cloud storage (e.g. not Glacier), then partial file restores may be done from the original file without indexing it first. This will cause more data to be read, but only the headers and tables necessary to find the audio/video/data the restoration needs

Scenario 4 - in line conversion

  • This assumes the basic processing in Scenario 1 has been done
  • For any restore scenario, the file restored can be a byte accurate re-wrap of the original into a new container, or it can be translated in process (on the fly) to any supported standard format. These formats include MXF Op1a, Op-Atom, P2, IMX, D11, IMF, MOV, Uncompressed and many other containers, with codecs including JPEG-2000, XDCam, MPEG-2, h.264, HEVC, AVCi 100/200, XAVC-S. XAVC, Long-G, TR-01, DV and many others
  • The commands can also be used with or without index files to convert all our part of local clips to any of these formats

PFR File Best Practices

Different workflows require different ways of saving and restoring files, but there are some general rules that can make it easier, especially when working with tape or other non sequential storage systems. Media files can be roughly broken down into a few categories:

  • Self contained, single files (MXF OP1a, MOV, AVI)
  • Multiple stream files (AVI+WAV, Avid OPAtom, MOV QT Reference)
  • Multiple files per stream (P2 MXF, XDCam MP4, Canon C300/700 MXF)
  • Sequences (DPX, TGA, TIFF)

If you are restoring from reasonable speed, random access devices, then all these types can be simply stored and retrieved as is. If there are sequential access, speed or cost issues, then it makes more sense to make each of these as easily accessible as possible before storing them.

Self Contained

These can be indexed and stored directly.


Multiple Stream Files

These can also be indexed and stored directly, as the RTIN can point at on file per stream. NetXCode automatically finds and joins the parts of the streams if it is a supported file type like Avid OPAtom, MOV reference, or video file with rationally named audio. If they are stored this way, there will be one chunk from each file that needs to be restored to make the output file, as the media is in separate files. This is handled in the getinoutbytes return as a series of temp names and start/end byte ranges. If you prefer a single chunk, please follow the guidelines in the Multiple Files Per Stream section.


Multiple Files Per Stream

The RTIN cannot describe streams that have multiple parts per stream. These are normally broken up to get around older disk format restrictions, often at 2 or 4 gigabytes. To deal with these, they should be pre processed (wrapped) to a self contained file like MXF OP1a before they are stored to tape. Using NetXCode to wrap them will cause the original audio/video to be copied to the new MXF without recompression and will automatically generate the RTIN as it is creating the new file. A proxy file can also optionally be created while rewrapping the file. This MXF now becomes the file you would restore from and that should be stored to tape.
NetXCode fully supports automatically joining the parts of most broadcast and post production files for playback and wrapping, including

  • Panasonic P2
  • IMF
  • AS-02
  • DCP
  • Canon C300/700
  • Sony XDCam MXF
  • Sony XDCam MP4
  • Avid OP-Atom
  • Grass Valley K2 Server Format
  • Multi card MXF capture


Avid OP-Atom – Special Case

NetXCode supports automatically joining Avid OP-Atom files and creating Avid OP-Atom files that can be directly dropped into the Avid bin for pickup on the next database refresh. If you are using this workflow, then then the Multiple Stream Files method works best. If you are importing files into Avid then you will not be able to use OP-Atom, as Avid cannot import even its own. It will want to see the same essence, but in an OP1a container. In this case, you should re-wrap the OP-Atom to OP1a before storing them. It is also possible to restore OP-Atom to OP1a using the “type mxf-op1a” when doing the restore, but if they are stored as OP-Atom you will still need to restore a chunk from each stream to PFR them.


Sequences

As sequences can be restored ‘per file’, they do not need to be indexed.


Sample Command Lines

Basic Conversion Re-Wrap

NetXCopy -s <sourcefile.gxf> -t <targetfile.mxf> -p wrap

Create New Sony XDCam 50

NetXCopy -s <sourcefile.mxf> -t <targetfile.mxf> -p mxf-OP1a-MPEG

Make a Basic Proxy with Audio Mix Down

NetXCopy -s <sourcefile.lxf> -t <proxyfile.mp4> -p mp4-h264 -stereo

Make a Specific Proxy

NetXCopy -s <sourcefile.gxf> -t <proxyfile.mp4> -p mp4-h264 -width 360 -height 202 -h26xprofile main -h26xlevel 41 -gopsize 60 -kilobitrate 800

Make an AS-11 DPP

NetXCopy -s <sourcefile.mov> -t <as11dpp.mxf> -p mxf-as-11-hd-dpp

Make a 10 Bit h.264 OP1a MXF

NetXCopy -s <sourcefile.dpx> -t <mxfop1ah264_10.mxf> -p OP1a_HBR_50

Create An Index For Offline/Tape File

NetXCopy -s <sourcefile.mxf> -p index

Partial File Restore From a Tape

Request the start and end to restore from tape for the actual restore

NetXCopy -s <sourceindex.rtin> -p getCopyInOut -in 01:00:10:00 -out 02:04:30:00 -alignment 4096 

This will return (tempFile=sourceindex_147894272_298274816.mxf, a start byte position u64in=147894272, and an end byte position u64out=298274816).  Copy the byte section specified from the source= to the tempFile= into your temp directory (d:\record\).  Then call the actual restore:

NetXCopy-s <sourceindex.rtin> -t <targetfile.mxf> -p getCopyInOut -in 01:00:10:00 -out 02:04:30:00 -alignment 4096 -tempFolder d:\record\

Replace Audio and Captions in an MXF File

NetXCopy -s <sourcefile.mxf> -t <targetfile.mxf> -p mxf-xdcam-1080i -cc <newlanguage.mcc> -afile <newlanguage.wav>

Make A JPG From A Video Frame

NetXCopy -pisrc <sourcefile.avi> -pidst <sourcefile_100.jpg> -piframe 100

Make a Series Of JPG Frames For Each Second

NetXCopy -pisrc <sourcefile.avi> -pidst <sourcefile_100.jpg> -piskip 30

Write an XMP Metadata File To Disk

NetXCopy -s <sourcefile.avi> -m

Get Clip Info From a Camera Card

NetXCopy -s <path-to-card> -p cardinfo -t 2


Command Line Parameters

netxcopy -s -t [-a ] -p [-in <00:01:00:00> -out <00:02:00:00> -fg]

-s - The source file name and path
-t - The target file name and path
-a - The ACK file name and path. This is the XML acknowledgement file made after a copy
-p metadata - Return XMP metadata for a media file
-p cardinfo - Return XML/JSON info on clips on a camera card
-p dir - Return XML/JSON directory listing
-p clipfiles - Return all the files associated with a media clip
-p md5 - Calculate or check the MD5 value source, send compare string as target
-p - Profile to use. Current profiles include:
    > copy - copy the whole file
    > wrap - re wrap file or part of a file
    > index - create an RTIndex for a file
    > getCopyInOut - get the extents required for a pfr, or use them with a temp file
    > mp3-128kbps - Audio MP3 file 
    > wave - Audio wave file
    > mov-YCbCr8Bit - QuickTime MOV 8 bit uncompressed YcbCr file
    > mov-dvcprohd - QuicktTime MOV DVCPro HD (1080/720)
    > mp4-h264 - MPEG-4 h264 AAC Audio
    > mxf-xdcam-720p - True XDCam MXF 8 channel audio
    > mxf-dvcprohd-720p - MXF DVCPro HD 720p
    > mxf-xdcam-1080i - True XDCam MXF 1080i 8 channel audio
    > mxf-dvcprohd-1080i - MXF DVCPro HD 1080i 29/25 fps
    > mxf-OP1a-MPEG - OpenMXF XDCam MPEG-2 16 channel audio
    > mxf-OP1a-h264 - MXF h.264
    > mxf-OP1a-HDF - MXF MPEG-2 HDF Standard
    > mxf-as-11-sd-pal-dpp - MXF AS-11 SD PAL DPP
    > mxf-as-11-sd-ntsc-dpp - MXF AS-11 SD NTSC DPP
    > mxf-as-11-hd-dpp - MXF DPP AS-11 AVCi HD
    > mov-proreshq - QuickTime MOV ProRes HQ
    > mov-proreslt - QuickTime MOV ProRes LT
    > mov-prores422 - QuickTime MOV ProRes 422
    > mov-prores444 - QuickTime MOV ProRes 444(4)
    > scaledown2000k - MP4 264 960x540, 2mbs, AAC
    > scaledown500k - MP4 264 480x272, 0.5mbs, AAC
    > hd1080-5000kbs - MP4 HD 1080 with a target bitrate of 5 mbs
    > hd720-2500kbs - MP4 HD 720p with a target bitrate of 2.5 mbs
    > hd360-1250kbs - MP4 HD 360p with a target bitrate of 1.25 mbs
    > h264-7500kbs - MP4 Any resolution with a target bitrate of 7.5 mbs
    > Proxy-h264-5000kbs - MP4 high quality proxy for web
    > LBR-h264-10000kbs - Low bit rate, high quality local MP4
    > mxf-OP1a-JPEG2K - Samma style JPEG2000 YCbCr
    > mxf-AS-02-h264-10 - 10 bit 50 Mbs h.264 in AS-02 MXF
    > DASH-MP4-Multibitrate - Multi bitrate MP4s with DASH files
    > HLS-TS-Multibitrate - Multi bitrate TS streams with M3U8 files
    > MP4-MultiOutput - Multi MP4 with optional burn in files
    > TS-TR-01-JPEG-2000 - TR-01 JPEG-2000 transport stream
    > TS-MPEG2 - MPEG-2 4:2:0/passthrough transport stream
    > TS-h264 - h.264 4:2:0/passthrough transport stream
    > OP1a_HBR_50 - OP1a MXF h264 4:2:2 10 bit
    > mp4-XAVC-S_4_2_0 - MP4 Sony XAVC-S 4:2:0
    > mp4-XAVC-S_4_2_2 - MP4 Sony XAVC-S 4:2:2
    > aces - ACES image files
    > MXF-RDD-25 - MXF RDD-25 Proxy
    > dnxhd-mxf-720p - DNxHD 720p 50, 59, 60
    > dnxhd-mxf-1080p - DNxHD 1080p 25, 29
    > dnxhd-mxf-1080i - DNxHD 1080i 25, 29
    > dnxhr-mxf-10-hq - DNxHR High Quality 10 bit
    > dnxhr-mxf-8-hq - DNxHR High Quality 8 bit
    > dnxhr-mxf-sq - DNxHR Standard Quality
    > dnxhr-mxf-lq - DNxHR Low Quality
    > amt3-HQX_10 - AMT 3 DNx HQX 10 Bit
    > amt3-HiQuality - AMT 3 DNx High Quality
    > amt3-StandardQuality - AMT 3 DNx Standard Quality
    > amt3-LowQuality - AMT 3 DNx Low Quality
    > amt3-DNxHD36 - AMT 3 1080 DNxHD 36
    > amt3-Consolidate - AMT 3 Any Avid Supported Codec
-type mxf-op1a -- the exact file type to write, otherwise auto
    > mxf-op1a - standard OP1a
    > mxf-sonyhd - Sony XDCam compatible
    > mxf-as02 - AS - 02 spec MXF
    > mxf-avid - Avid OP-Atom (Drastic->bin)
    > mxf-amt = Avid OP-Atom (Avid->aaf)
    > mp4-fmp4 - Fragmented MP4(normal MP4 if not set)
    > mov - QuickTime MOV
-in <00:01:00:00> - the starting point for the output file in time code or absolute position
-out <00:02:00:00> - the ending point for the output file in time code or absolute position
-absin <200> - the absolute (zero based) start time for the output file (overrides -in)
-tcoffset <00:01:00:00> - Offset the timecode by this amount
-tc <01:00:00:00> - Replace the output timecode starting with this timecode
-ub - Replace the output userbits with these userbits
-absout <400> - the absolute (zero based) end time, exclusive, for the output file (overrides -out)
-width - output width of the video (only for arbitrary codecs like h264, hevc and prores)
-height - output height of the video (in not set, the input size or codec size will be used)
-copy - make a copy of the file section we need, instead of reading directly
-dest - folder or folder and file name for the temp file when using copy
-dest - folder or folder and file name for the temp file when using copy
-cc - replacement closed caption file>
-afile - replacement source audio tracks
-v - replacement source video track
-stereo - force a stereo pair (mix down) output
-aroute <12345678> - route channels to specific outputs
-uuid - override the UUID of the file with this one
-kilobitrate - override the kilo bit rate
-h26xprofile - override the profile type
-h26xlevel <51> - override the level
-encodemode <0 / 1> - 0 normal, 1 fastest
-gopsize <15> - size of encoded gop
-tempfolder - Temporary folder to store partial file
-alignment - Alignment value for any temporary partial files, for GetCopyInOut profile
-flags flag - Extra flags for special operations
  frameaccurate,notifyemam,testmode,timeconversion,proxyenable
-m - Save the metadata in an XMP file
-fg - force the GUI on
-fc - force command line
NOTE: the parameters in [square brackets] are optional.

For JPEG picons
-pisrc - source fot the
-pidst - target folder and name
-pisize - size of picon, 100%
-piframe - frame to use to make the picon
-piskip - if set, make a picon of each frame at this distance for the whole file
-width - output width of the picon image