man.scrbl

#lang scribble/base

@(require scribble/manual
          (for-label racket/base
                     video/base))

@title{Video Language Overview}

Racket's @racket[video] package is a DSL for Non-linear
video editing. Similarly to @racket[scribble] and
@racket[slideshow], you can create videos while taking
advantage of the full Racket language.

@section{Hello Green}

Create a file @tt{"green.vid"} with this content:

@codeblock{
 #lang video

 (color "green")}

The first line, @code{#lang video}, indicates that the file
implements a video. The content of the file, @code{(color
 "green")} indicates that the video should be a solid green screen.

You can use the @exec{raco video} program to render the video:

@itemlist[
 @item{Run:
  @commandline{raco video green.vid}
  to see a preview of the video. This includes playback
  controls and the rendered video. In this case the
  preview will be a solid green frame.}
  
 @item{Run:
  @commandline{raco video -o green.mp4 green.vid}
  to render the video to an MP4 file.}

 @item{Run:
  @commandline{raco video -t xml -o green.xml green.vid}
  to render the video to a plain text XML file.}]

@section{Multiple clips}

A video is formed by creating a
playlist of clips. All top-level video clips in a file are
placed in a single playlist. For example:

@codeblock{
 #lang video

 (color "blue" #:length 50)
 (color "green" #:length 50)}

Creates a movie with 50 frames of blue (about 1 second),
followed by 50 frames of green.

You can even constructs native to, such as lists, to create a playlist:

@codeblock{
 #lang video

 (for/list ([i (in-range 255)])
   (color (list 0 i 0)))}

Creates a clip that slowly fades from black to green. We
could use @racket[playlist] to convert the playlist to a
clip, but this is not needed, because playlists are clips.

You can even include clips generated by other video language
programs:

@codeblock{
 #lang video

 (include-video "green.vid")
 (include-video "blue.vid")
}

This play the @tt{"green.vid"} program from above, followed
by a similar @tt{"blue.vid"} program.

@section{Creating pictures}

Using the @racket[picture] function, images created with the
@racketmodname[pict] and @racketmodname[racket/draw]
libraries can be placed in videos. For example, the
following program creates a video of the standard fish:

@codeblock{
 #lang video
 (require pict)
 (picture (standard-fish 500 500))
}

Notice that @racket[(require pict)] was placed in the file
without adding a clip to the video. Only top level
expressions add clips to the video, definitions and require
forms do not. We could further extend this idea to create a
movie of a standard fish swimming in progressively contaminated water:

@codeblock{
 #lang video
 (require pict racket/draw)
 (define fishy (standard-fish #:color "chocolate"
                              #:width 500
                              #:height 255))
 (for/list ([i (in-list 255)])
   (picture (cc-superimpose (filled-rectangle #:color (make-object color% 0 i (- 255 i))
                                              #:width 600
                                              #:height 600
                            fishy))))
}

@section{Importing camera videos}

While creating videos from clips is fun, we need to be able
to import videos from other sources, such as a camera. This
is be done with @racket[clip]:

@codeblock{
 #lang video
 (clip "step.mp4")
}

Here, @tt{"step.mp4"} is a video of a person walking. We can
use @racket[clip]'s optional arguments to keep only a slice
of the clip, play it slower, and even backwards. Here, we
make this person take the step several times before letting her continue on:

@codeblock{
 #lang video 
 (define step "step.mp4")
 (clip step #:out 100)
 (for/list ([i (in-range 50)])
   (clip step
         #:in 100
         #:out 200
         #:speed (if (even? i) 1 -1)))
 (clip step #:in 100)
}

@section{Applying Filters}

Every clip in @racket[video] can have a filter attached to
it, using @racket[attach-filter]. Filiters alter the
behavior of a clip such as turning it to grayscale,
inverting the colors, or even adding a watermark. As an
example, we can create a grayscale version of the girl
walking using the @racket[grayscale-filter]:

@codeblock{
 #lang video
 (define walking (include-video "walking.vid"))
 (attach-filter walking (grayscale-filter))
}

@section{Clip Properties}

Let's say we want to add a watermark to our walking video.
However, we want to add it to the first 10% of the video,
and then we want the watermark to go away. Every clip has a
set of properties attached to it. The relevant one here is
@racket['length], which stores the clip's length. We can use
@racket[properties-ref] to get the property:

@codeblock{
 #lang video
 (define walking (include-video "walking.vid"))
 (define walking-length (properties-ref walking 'length))
 (attach-filter walking (watermark-filter 0 0 1/8 1/8
                                          #:in 0 #:out (/ walking-length 10)))
}

@section{Using Multitracks}

While playlists are powerful for laying out clips
horizontally---playing clips in succession. Videos also need
an abstraction for laying out clips vertically---playing
clips at the same time. For example, we may want to play a song with our walking clip:

@codeblock{
 #lang video
 (multitrack
  (include-video "walking.vid")
  (audio "music.wav"))
}

The @racket[audio] function is similar to @racket[clip], but
the created clip only contains audio. @racket[multitrack]
superimposes a list of clips on top of each other. Like
@racket[playlist], a @racket[multitrack] is also itself a
clip. In the top clip only plays audio, the walking
clip plays in lockstep with the music.

This example works only because @tt{"music.wav"} does not
contain any video. If it did, @tt{ "walking.vid"} would be
completely overshadowed. We can tell @racket[multitrack] how
we would like the videos to be mixed by taking advantage
that each clip is @racket[eq?] with itself. The following
clip plays the walking clip and the toxic fish clip in lockstep,
with the fish clip taking up the top-left corner of the screen:

@codeblock{
 #lang video
 (define walking (include-video "walking.vid"))
 (define polluted-fish (include-video "polluted-fish.vid"))
 (multitrack
  walking polluted-fish (audio "music.wav")
  #:effects (list (composite walking polluted-fish 0 0 1/4 1/4)))
}

Another, more concise way, to express this is:

@codeblock{
 #lang video
 (define walking (include-video "walking.vid"))
 (define polluted-fish (include-video "polluted-fish.vid"))
 (multitrack
  walking
  (composite 0 0 1/4 1/4)
  polluted-fish
  (audio "music.wav"))
}

Here, @racket[composite] pulls the top and bottom tracks
from the ones above and below it in the multitrack.

@section{Transitions}

Jumping from one clip to the next can sometimes be jarring.
Because of this, @racket[video] also includes transitions
which can combine two clips together.

For example, we can make a transition of a clip of a person
walking, with an equivalent one that is grayscale:

@codeblock{
 (define walking (include-video "walking.vid"))
 (cut-clip walking #:out 100)
 (fade-transition 50)
 (attach-filter (cut-clip walking #:in 100)
                (grayscale-filter))
}

Here, @racket[fade-transition] takes 50 frames from the
first clip, and 50 frames from the second clip, and inserts
a new clip that lasts 50 frames that fades from the first
clip to the second. Note here that the resulting playlist
will now have three rather than two clips, and will be 50
frames shorter.

We also introduce @racket[cut-clip], which takes itself a
clip, and returns the same clip, except cut between
@racket[#:in] and @racket[#:out].

@section{Abstractions}

So far, all of our programs have used @code{#lang video}
directly. But remember that each of these primitives produce
videos themselves. We can use the linguistic constructs in
Racket to build new videos, similarly to the
@racketmodname[pict] library. We can get all of these
primitives @code{#lang video} provides with
@racketmodname[video/base].

As an example, we create a function that takes an arbitrary
video, and places a watermark on the first 5% of the video frames.

@racketblock[
 (code:comment "; utils.rkt")
 (require video/base)
 (define (add-watermark vid)
   (attach-filter
    vid
    (watermark-filter 0 0 1/8 1/8
                      #:in 0 #:out (/ (properties-ref vid 'length) 10))))]

And a video that uses this file:
@codeblock{
 #lang video
 (require "utils.rkt")
 (add-watermark (include-video "walk.vid"))
}