This is a sample of the many pages of documentation I have written.

-------------------------------------

Member of the Product Management team, sales, and many customers have requested a better tool for Matte Painting. In addition, features that contribute to “Finishing” functionality have been requested such as roto-scoping, motion tracking, image stabilization, easy wire removal, video paint, onion skinning, etc.

A digital painting tool can encompass a huge amount of functionality and in general is quite open ended. In the case of a tool for 844/X, “Paint Tool” functionality can be roughly be broken into two types along the primary usage: fixing up footage and creative content creation. This spec addresses the first type, with hopes of implementing many creative features as well.

After a market search, it is felt at this time that the Curious gFx Pro paint tool from Curious Software best meets our needs for functionality and ability to integrate into 844/X.  They also seem to be on a schedule to provide timely delivery for Elise.

In addition to painting on video and matte clips, an appropriate external editor could modify audio waveforms or still clips. Hence we include in this design a description of a generalized External Editor  feature to be added to 844/X.

The New Paint Tool (NPT) is a third party application integrated into 844/X so as to provide as seamless a video painting experience as possible for a standalone application. In addition to painting, the NPT will provide motion tracking and stabilization, roto-scoping, and more.

The unit of work with which to access the NPT is a clip or clip instance encapsulated in an Edit Job. Users will right click on a clip in 844/X and select editing options to launch the NPT. The NPT will then pop to the front, the user will be able to edit/paint/track, and then close the edit job. At this time 844/X will incorporate the resultant clip back into the bin or timeline from whence it was launched.

The NPT  implementation will consist of 844/X engineering efforts as well as the third party efforts. An interface will be defined describing how the two applications will coordinate and communicate to get clips edited.

While the NPT we have selected has a good UI and most of the minimum paint functionality we need, Media 100 will assemble a list of UI and functional enhancements we think should be added to the NPT.

A schedule has been developed that consists of many milestones and drops from both parties to ensure we are on the correct development track.

See Appendix A for detail requirements we are looking for in a third party paint tool.

The features we expect to gain through a NPT can be used at all stages of editing, compositing, and finishing. That is, all users can benefit from this feature.

Some users will be looking only to repair and improve footage such as stabilization, and some will be looking to do artistic work such as painting.

Users will benefit from the improved work flow provided by a tight integration with a third party tool providing all the paint-like features we are looking for.

The interface is in two parts: that in 844/X needed to launch NPT and deal with results, and the Paint Tool interface itself. The Curious gFx UI can be described as Inscriber-like, but a complete description is beyond the scope of this document. We will describe the 844/X side of NPT, however.

NOTE

This document assumes the NPT will not have a re-acquire ability and that all media it produces is fragile. If that changes, some of the precautions we take because of this will not have to be observed.

Clip Types

We intend to allow painting on any clip type except color clips.  Issues arise when painting on some clips because in a timeline, the target clip instance gets replaced with the NPT output clip, and the original can get lost.

The editing process for all 844/X clip types with the NPT is:

·        Media backed clips – Provide the backing media file to the NPT. The user paints on a temporary media file; that file is then quietly  imported into 844/X as a new clip with a possible clip instance replace being done. The created media file is fragile media. (That is, until the NPT can generate and use rebuild info.)

·        Color clips – We will not allow painting on color clips for the first release.

·        Still clips – The user will have the option to specify whether to paint on stills as a static (producing another still as output) or as animated (producing a full clip length motion media backed clip.)

·        Subcomp – Users will have an option of saving a copy of the subcomp in a bin and editing the media file for the subcomp, or of opening the subcomp, and creating a paint clip to go in the top layer of the subcomp.

·        CG clip – When the user edits a CG clip in the paint tool, they will have a chance to create a copy of the CG clip in a bin before editing. After editing, 844/X will proceed as for media backed clips. Essentially we will promote (demote?)  the CG clip to a paint clip while optionally saving a copy of the CG clip. Also, users will also have the option of putting the CG clip into a subcomp and creating a paint clip in the top layer of that subcomp.

·        Paint clip – As for CG clips, users will have an option of saving the original in a bin.

The clip part flow will be like the existing Matte Tool. For example, editing  the video of a clip, say C1, that has {V,M,A1,A2} parts will produce a clip, C2, with only the edited video part. For a subsequent release, we may support an aggregate clip approach where C2 will get the non edited parts from C1 and thus be identical except the part that was painted on.

NOTE

A very important difference between the old matte tool (OMT) and the NPT is that the OMT can edit strokes and shapes of an existing clip because it has an object oriented, vector based editing interface. That is, part of the clip information saved to disk is the list of painting operations that performed.

The Curious gFx Pro NPT does not have this ability and is a flat file pixel–painter based editor. When the NPT reedits a painted clip, users will not be able to get at previous strokes. All they will see is pixels, no strokes.

This is why the overall flow is to by default preserve the existing clip users will paint on. Generations of paint clips will build up allowing users to revert back to any of them, assuming they have not removed the media.

This is also why the existing Matte Tool will be kept in 844/X.

Bin Clip Editing Process

A bin clip can be edited with NPT in a way similar to that of the current matte tool. A new color or matte clip can be created, or an existing one can be painted over. Additionally, tracking data can be generated from an existing clip or matte and saved as a DVE filter stack.

Some bin context (RMB) menu items will be added or modified:

·        Create Matte Clip… –  A dialog will let users choose the old or new paint tool, static or animated matte, clip duration, 

·        Create Paint Clip… – A dialog will let users choose between static or animated matte, specify a clip duration,  provide a background clip for context (select another clip from the same bin),

·        Motion Tracking: Stabilization – The NPT will open with the target clip, the user will do the tracking, close the NPT, and a filter stack will be created named from the clip name (e.g. Speedboat_stabalized).

·        Motion Tracking: 1 Point Pin – The NPT will open with the target clip, the user will do the tracking, close the NPT, and a filter stack will be created named from the clip name (e.g. Speedboat_trackdata).

·        Motion Tracking: 4 Point Pin – The NPT will open with the target clip, the user will do the tracking, close the NPT, and a filter stack will be created named from the clip name (e.g. Speedboat_4PointTrack).

·        Edit Matte… – A dialog will let users choose between the NPT or old one and whether to replace the old matte clip or make a copy.

·        Edit Clip… – Display a clip type dependant dialog to choose between the options described in the Clip Type section above.

·        Edit Clip & Matte… – Edit both video and matte. The user can paint on either or both clip parts. A combinations of options from above will be in a dialog to control this job.

The typical Bin clip based NPT editing process is:

·        RMB on the clip (C1) to select “Edit Clip…”

·        A dialog will allow the user to choose options that control the edit job.

·        NPT will be loaded into memory if it’s not already

·        The C1 media  file will be passed to NPT, and NPT will be activated and come to the front of the screen

·        The user will paint on C1

·        The user will select a NPT menu item such as “Close and return to 844/X”

·        844/X will (through directory monitoring and an IPC JobDone message) notice the new media file and do the following:

o       Quietly import the media (no Import dialog), creating a new clip for that media

o       Put the new clip in the same bin, synthesizing a name for it from the original clip name if the user did not specify an output clip name.

Timeline clip instance editing process

The editing process is similar from the timeline except the user can select whether all instances of the clip part being edited should be updated, or just the one from which NPT was launched.

There will be two types of motion tracking workflows:

·        Motion Tracking: Stabilization – The NPT will open with the selected clip instance (clip thereof), the user will do the tracking, close the NPT, and a DVE be created to apply the stabilization keyframes and added to the selected clip instance’s filter stack.

·        Motion Tracking: 1 and 4 Point Pin – The user will RMB on the clip instance to which the tracking data will be applied (destination instance) and select in a dialog the target clip in which to do the tracking (source clip).  On NPT exit, the destination instance’s DVE will be modified using the tracking data (the details of the DVE modification are <tbd>.

Clip instances will be edited by passing the clip they are an instance of to the NPT. Users will see the clip without any filters applied to the instance.

If the user requests a background and/or foreground clip in the new Paint Dialog, depending on options, 844/X may cook down the tracks below and above the target clip instance to provide these media files.

Background options in the Paint Dialog will be:

·        Provide just the instance(s) in the track immediately below the clip,

·        Use the clips on the ‘on’ tracks below the target clip instance and cook them

Foreground clip options will be:

·        Use only the clip(s) immediately above the target clip,

·        Use all the clips on the ‘on’ tracks above the target clip.

In addition to these options, there will be many more to control the painting process, the details of which are <tbd>. For example, when launched from the timeline for painting, options will exist in the Paint Dialog to create a copy of the result and of the original clip in a bin.

A number of new or modified context menu items similar to those in the Bin section above will be added to the timeline(only one operation can be selected at a time):

·        Blank track with range selected – Users will be able to Create a matte or paint clip of (out – in) duration.

·        Existing clip with matte – Options to edit the matte with the new or old tool, or create motion tracking data.

·        Existing clip without matte – Options to paint the clip, create a matte, or create motion tracking data.

There will be some clip type dependent processing:

·        CG instance – Users will always be prompted to save the original CG clip into a bin before the timeline clip and instance are “promoted” to a paint clip.

·        Subcomp instance – Users will always have the option of opening up the subcomp and placing a paint clip in it on the top layer, or of saving the subcomp to a bin before demoting it to a paint clip.

When the NPT is launched from the timeline and there is sufficient backing media, trim handles of <some default length> or as specified in the Paint Dialog will be created.

Editing both video and matte will probably be allowed. That is, the user will be able to RMB of an instance with matte, and paint on either or both when in the NPT. Upon NPT exit, both will be incorporated back into the timeline.

Media

All NPT output files are fragile media until the third party can support creation and processing of reacquire  files  (‘strokes’ data or rebuild info).

All NPT output files will be stored in a “FragileMedia” folder.

The NPT cannot change the duration of a clip for phase one, except that it can accept a still as input and produce moving video/matte as output provided the output duration is the same as the input still.

There will be an app wide preference for whether to provide and specify how long  trim handles for each NPT call should be.

When a proper clip and media database is implemented for 844/X and when the NPT supports rebuild info, some of the problems that force always saving original clip contents will be solved.

Even though the NPT may be able to paint on both matte and video for a given UI level operation, it will still need to write separate video and matte media files upon job completion. These can then be imported by reference back in 844/X.

Clipboard Usage

844/X and the NPT will use the system clipboard to pass many types of data between each other.

Pixel images

844/X will provide an ability to copy the full frame of a subsampler’s current image to the system clipboard in a bitmap format.

844/X will provide a new Paste option in a bin to paste a bitmap image (such as from above) as a single frame clip into the bin. This will be a fragile data clip.

Curve Data

Currently 844/X copies curve data to the clipboard in a proprietary format that assumes the pasting code is in the same process as the copying code.

For the initial implementation, 844/X will not provide a means of transferring a KPE curve to the NPT.

Tracking Data

Motion tracking data keyframes from the NPT can be stored on the system clipboard (for a copy/cut command) in a Curious defined format such that 844/X can paste it.

844/X will continue to support pasting AE key frame data.

Tracking data on the clipboard will be available to paste in a Bin as a filter stack as can be done now for AE keyframes.

Clips and Clip Instances

844/X will not put a public format on the clipboard for clips and clip instances. This might be revisited for a subsequent release .

Other factors

The NPT will have to / want to use the QuickTime Video Out facility and our Instant media codec. NPT will have to learn to release Vout when it does a long render and maybe even every time it is put in the background or minimized. That way, 844/X can re-gain control of the hardware when it goes to the foreground.

For release two and maybe release one, we will want to store ‘rebuild’ info from NPT. That is, we need to be able to recreate/re-acquire a clip given enough information for the NPT to do all the drawing on the original file again. It is not clear at this point if the NPT will be able to support this.

As was mentioned, the notion of an “External Editor” is just a small generalization away from a New Paint tool, so it will be included in this project. It will be implemented on a “time available” basis.

Users define a third party media editing tool in an 844/X preferences dialog, and then launch that tool from a context menu on a clip or clip instance.

Defining an External Editor

There will be a new preferences tab that allows users to specify external editors for each clip type 844/X supports. For each type users will be able to specify:

·        The name of the editor (including the full file path)

·        The command line with which to invoke the editor, including the ability to specify command line options (e.g. “-edit”) and parameters (e.g. $input) to represent the original target clip and the output clip on the command line.

·        Processing options such as always make a copy in <some bin> before editing.

Launching an External Editor

When a user right clicks on a clip or clip instance, 844/X will choose an editor from the defined list and provide a menu entry for the user to launch the external editor. When the edit is completed, 844/X will perform the clip / clip instance replacement as described earlier.

When 844/X launches the external editor, it will create a QuickTime reference movie with as much clip information as is provided to the NPT. This way, if the editor supports reading that information (e.g. markers), it will be available.

In an effort to streamline all paint related workflows, Media 100 would like to integrate as much as possible the UI of a NPT’s individual functions (e.g. tracking, painting, etc.) Some features might be implemented as small tools, others as filters, and others as concepts new to 844/X (e.g. a Tracker object).

A standalone third party paint tool is a good first pass at solving many problems 844/x has at this time. There are many reasons (such as editing a timeline clip instance will not show the instance’s filters) to breakdown the aggregate NPT feature set into smaller modules and more completely integrate each with the 844/X editing model. We need to plan now for this eventual tight integration.

Send mail to paul at sign softwaretool.com with questions or comments about this web site.