Net-X-Copy - Partial File Restore

verifycc icon

Net-X-Copy is part of the Net-X-Convert/Net-X-Proxy workstation and Net-X-Code server software. It is designed to convert audio/video files from one type to another and to perform partial file restores from indexed and non indexed files on disk, tape or cloud. This article contains the Linux/OS-X/Windows command line API for this tool.  A RESTful API is also available.

Net-X-Copy - Partial File Restore

Net-X-Copy supports partial file restore for all the standard broadcast and postproduction formats we support without the need for a mezzanine or intermediate file format.  The original files are indexed, a timecode accurate proxy is created, and then any section of the original file can be recalled via timecode or absolute offsets based on HTML5 playback of the proxy.  The high resolution asset can be stored on near line storage, cloud storage (HTTPS/FTP/Amazon/Google/SMB), tape or any other kind of block, object or file storage your server can access.

The index/proxy/restore can be controlled via our RESTful API in www.net-x-code.com or via the command line parameters descibed below.


Typical Scenarios

This section has some typical proxy, convert and partial file restore workflows. The first scenario is a complete list. The subsequent scenarios assume the first few steps in the first scenario have already been accomplished

Scenario 1 - full access/on line source

  • The file (MXF, MOV, AVI, CINE, etc) arrives at ingest
  • A “command=copy&profile=index” command is sent to index to original filename
  • A “command=copy&profile=mp4-h264” command is sent to make a proxy file with time code, multitrack audio, closed captions, metadata and proxy index filename
  • Two “command=copy&pisrc&pidst” commands are sent to create JPEG images for the source and proxy files
  • At this point, the proxy and main index can be stored in a real or near real time storage, and the main file may be moved to long term storage, tape, cloud (google/s3) or other offline storage
  • The user uses the HTML5 player’s time code (or other time code source) to set one or more In and Out points on the file that needs to be restored
  • A “command=copy&profile=wrap” is sent to access the bytes of the original file and create a new file of the same type, without any recompression of audio or video, at the target location

Scenario 2 - tape restore

  • This assumes the basic processing in Scenario 1 has been done
  • A “command=getcopyinout” is sent with the absolute or time code based in and out points, and the index of source file it will come from
  • This returns a series of one or more file names with start and end byte locations
  • At this point, the controller restores those byte areas of the files to the name specified by the return
  • Once the areas are restored, a “command=copy&profile=wrap” is sent along with the temp folder to create the new output filename

Scenario 3 - cloud restore

  • This assumes the basic processing in Scenario 1 has been done
  • If the index file is stored on cloud, it can be restored locally first, or read directly from the cloud (https, ftps, aws)
  • If the main file is in Glacier, then a command will be sent to restore the section needed to S3 before the restore is done
  • Once there is access to the file, or file part, the “command=copy&profile=wrap” can be called normally
  • If the resource is in available cloud storage (e.g. not Glacier), then partial file restores may be done from the original file without indexing it first. This will cause more data to be read, but only the headers and tables necessary to find the audio/video/data the restoration needs

Scenario 4 - in line conversion

  • This assumes the basic processing in Scenario 1 has been done
  • For any restore scenario, the file restored can be a byte accurate re-wrap of the original into a new container, or it can be translated in process (on the fly) to any supported standard format. These formats include MXF Op1a, Op-Atom, P2, IMX, D11, IMF, MOV, Uncompressed and many other containers, with codecs including JPEG-2000, XDCam, MPEG-2, h.264, HEVC, AVCi 100/200, XAVC-S. XAVC, Long-G, TR-01, DV and many others
  • The commands can also be used with or without index files to convert all our part of local clips to any of these formats

Sample Command Lines

Basic Conversion Re-Wrap

NetXCopy -s <sourcefile.gxf> -t <targetfile.mxf> -p wrap

Create New Sony XDCam 50

NetXCopy -s <sourcefile.mxf> -t <targetfile.mxf> -p mxf-OP1a-MPEG

Make a Basic Proxy with Audio Mix Down

NetXCopy -s <sourcefile.lxf> -t <proxyfile.mp4> -p mp4-h264 -stereo

Make a Specific Proxy

NetXCopy -s <sourcefile.gxf> -t <proxyfile.mp4> -p mp4-h264 -width 360 -height 202 -h26xprofile main -h26xlevel 41 -gopsize 60 -kilobitrate 800

Make an AS-11 DPP

NetXCopy -s <sourcefile.mov> -t <as11dpp.mxf> -p mxf-as-11-hd-dpp

Make a 10 Bit h.264 OP1a MXF

NetXCopy -s <sourcefile.dpx> -t <mxfop1ah264_10.mxf> -p OP1a_HBR_50

Create An Index For Offline/Tape File

NetXCopy -s <sourcefile.mxf> -p index

Partial File Restore From a Tape

NetXCopy -s <sourceindex.rtin> -t <targetfile.mxf> -p wrap -in 01:00:10:00 -out 02:04:30:00 -alignment 4096 

Replace Audio and Captions in an MXF File

NetXCopy -s <sourcefile.mxf> -t <targetfile.mxf> -p mxf-xdcam-1080i -cc <newlanguage.mcc> -afile <newlanguage.wav>

Make A JPG From A Video Frame

NetXCopy -pisrc <sourcefile.avi> -pidst <sourcefile_100.jpg> -piframe 100

Make a Series Of JPG Frames For Each Second

NetXCopy -pisrc <sourcefile.avi> -pidst <sourcefile_100.jpg> -piskip 30

Write an XMP Metadata File To Disk

NetXCopy -s <sourcefile.avi> -m

Get Clip Info From a Camera Card

NetXCopy -s <path-to-card> -p cardinfo -t 2


Command Line Parameters

NetXCopy -s <sourcefile> -t <targetfile> -p <profile> [-t <type> -a <ackfile>] [-in <00:01:00:00> -out <00:02:00:00> -absin <00:01:00:00> -ansout <300> -width # -height # -copy -dest <folder> -cc <mcc/scc file> -afile <audio-file> -v <video file> -stereo -uuid <uuid-string> -kilobitrate # -h26xprofile # -h26xlevel # -encodemode # -gopsize # -tempfolder <temp Folder> -alignment # -m -fg -fc -pisrc <picon source> -pidst <picon dest> -piframe # -piskip # -width] -s <sourcefile> -- The source file name and path

  -t <targetfile> -- The target file name and path
  -a <ackfile> -- The ACK file name and path. This is the XML acknowledgement file made after a copy
  -p <profile> -- Profile to use. Current profiles include:
* copy - copy the whole file
* wrap - re wrap file or part of a file
* index - create an RTIndex for a file
* mp3-128kbps - Audio MP3 file
* mov-YCbCr8Bit - QuickTime MOV 8 bit uncompressed YcbCr file
* mov-dvcprohd - QuickTime MOV DVCProHD
* mp4-h264 - MP4 h.264 AAC Audio
* mxf-xdcam-720p - Sony XDCam 50 at 720p
* mxf-dvcprohd-720p - Op1a DVCPro HD 720p
* mxf-xdcam-1080i - Sony XDCam 50 at 1080i
* mxf-dvcprohd-1080i - Op1a DVCPro HD 1080i
* mxf-OP1a-MPEG - MXF Op1a MPEG-2 50 16 Ch Audio
* mxf-OP1a-h264 - MXF Op1a h.264
* mxf-OP1a-HDF - MXF Op1a HDF MPEG-2
* mxf-as-11-sd-pal-dpp - MXF AS-11 SD PAL DPP
* mxf-as-11-sd-ntsc-dpp - MXF AS-11 SD NTSC DPP
* mxf-as-11-sd-dpp - MXF IMX DPP AS-11
* mxf-as-11-hd-dpp - MXF AVCi 100 DPP AS-11
* mov-proreshq - QuickTime MOV ProRes HQ
* mov-prores422 - QuickTime MOV ProRes 422
* mov-prores444 - QuickTime MOV ProRes 444(4)
* scaledown2000k - MP4 264 960x540, 2mbs
* scaledown500k - MP4 264 480x272, 0.5mbs
* hd1080-5000kbs - MP4 - HD 1080 with a target bitrate of 5 mbs
* hd720-2500kbs - MP4 - HD 720p with a target bitrate of 2.5 mbs
* hd360-1250kbs - MP4 - HD 360p with a target bitrate of 1.25 mbs
* h264-7500kbs - MP4 - Any resolution with a target bitrate of 7.5 mbs
* Proxy-h264-5000kbs - MP4 high quality proxy for web
* LBR-h264-10000kbs - Low bit rate, high quality local MP4
* mxf-OP1a-JPEG2K - Samma style JPEG2000 YCbCr
* mxf-AS-02-h264-10 - 10 bit 50 Mbs h.264 in AS-02 MXF
* DASH-MP4-Mutibitrate - Multi bitrate MP4s with DASH files
* HLS-TS-Mutibitrate - Multi bitrate TS streams with M3U8 files
* TS-TR-01-JPEG-2000 - TR-01 JPEG-2000 transport stream
* OP1a_HBR_50 - OP1a MXF h264 4:2:2 10 bit
* mp4-XAVC-S_4_2_0 - MP4 Sony XAVC-S 4:2:0
* mp4-XAVC-S_4_2_2 - MP4 Sony XAVC-S 4:2:2
  -type mxf-op1a -- the exact file type to write, otherwise auto
* mxf-op1a - standard OP1a
* mxf-sonyhd - Sony XDCam compatible
* mxf-as02 - AS - 02 spec MXF
* mp4-fmp4 - Fragmented MP4(normal MP4 if not set)
  -in <00:01:00:00> -- the starting point for the output file in time code or absolute position
  -out <00:02:00:00> -- the ending point for the output file in time code or absolute position
  -absin <200> -- the absolute (zero based) start time for the output file (overrides -in)
  -absout <400> -- the absolute (zero based) end time, exclusive, for the output file (overrides -out)
  -width <width> -- output width of the video (only for arbitrary codecs like h264, hevc and prores)
  -height <height> -- output height of the video (in not set, the input size or codec size will be used)
  -copy -- make a copy of the file section we need, instead of reading directly
  -dest -- folder or folder and file name for the temp file when using copy
  -dest <folder> -- folder or folder and file name for the temp file when using copy
  -cc <mcc/scc file> -- replacement closed caption file>
  -afile <path-audio-file> -- replacement source audio track<s>
  -v <path-video-file> -- replacement source video track
  -stereo -- force a stereo pair (mix down) output
  -uuid <uuid string> -- override the UUID of the file with this one
  -kilobitrate <kbit rate> -- override the kilo bit rate
  -h26xprofile <baseline / main / high / high10 / high422 / high444> -- override the profile type
  -h26xlevel <51> -- override the level
  -encodemode <0 / 1> -- 0 normal, 1 fastest
  -gopsize <15> -- size of encoded gop
  -tempfolder -- Temporary folder to store partial file
  -alignment -- Alignment value for any temporary partial files, for GetCopyInOut profile
  -m -- Save the metadata in an XMP file
  -fg -- force the GUI on
  -fc -- force command line
NOTE: the parameters in [square brackets] are optional.

For JPEG picons
  -pisrc <file-to-picon> -- source fot the
  -pidst <where-to-make-picon> -- target folder and name
  -pisize <size> -- size of picon, 100%
  -piframe <frame-offset-into-file> -- frame to use to make the picon
  -piskip <number - of - frames - to - skip> -- if set, make a picon of each frame at this distance for the whole file
  -width <width> --output width of the picon image