Download and installation

Download the Droste effect filter for After Effects and Photoshop CS4
See the Pixel Bender Toolkit page on Adobe labs for details on getting PBT up and running on your machine.

After Effects: Just put the Pixel Bender .pbk filters in a folder within the Plug-ins/Effects/ directory of your installation then it will be available from the Effects menu.
Photoshop: See the release notes for installing the Pixel Bender plugin.

Quickstart

These are the steps you should follow for best results:

1. Set the size to match the source image.
2. Adjust the centerShift so that the spiral is centered on the area you want to repeat.
3. If the source image is a 24-bit PNG with a transparent centre section (like a picture frame) then set enableTransparencyInside to 1.
   If the image has a transparent outside (like a flower cut-out) set enableTransparencyOutside to 1
4. Adjust the innerRadius in combination with tweaking the centerShift values until the repeated section aligns nicely.
5. Adjust the center parameter to frame the overall image.

Basic settings:

Periodicity: sets the number of times the image is repeated on each level of the spiral. For picture frame type images leave this as 1. For circular images small adjustments might be needed to get a nice repeating spiral but it is always worth trying the setAutoPeriodicity setting.
The strand setting is the number of arms on the spiral. When set greater than 1 and applied to an image like a flower that has an outside transparency then strandMirror for seamless tiling.

Basic settings: 0 strands (classic Droste), 1 strand, 2 strands, single strand with increased periodicity

Advanced mapping options:

Increase the startLevel so that the image fills the outer area. Increase the numberOfLevels until the centre of the spiral has filled in.
When enablePoles is set then the polarCoordinates can be used.
When tilePoles is set then enableHyperDroste setting can be applied and the fractal points setting will then take effect. The untwist option will tile the untwisted annular section effectively unrolling a circular image.

Advanced settings: untwist, enable poles, enable poles with increased periodicity and hyper droste.

Tutorials

There is a very nice video podcast on using the filter over at Creative Cow thanks to Carl Larson.