gg

Web Attachments Module

Specify Software Project Staff
11 November 2013
Version 2.0

The 'Web Attachments Module' for Specify enables it to interact with a web server for reading, writing and deleting attachments. It is designed to enable the most flexibility while making it extremely simple for users to configure.

How It Works

The user enters a single URL string into a preference that indicates where Specify should look to download an XML file containing the URL templates for interacting with the Attachment Web Server (AWS) for performing all the necessary I/O functions. Each URL in the XML is a template with a set of pre-defines symbols that are replaced with values from the database so the AWS knows where the attachment come from and how it might be stored. The XML file is not cached on the user's local machine. This means the cache can move locations without the user having to re-configure Specify.


Here is an example of the XML file's contents:
<?xml version="1.0" encoding="UTF-8"?>
<urls>
    <url type="read"><![CDATA[http://localhost/cgi-bin/fileget.php?type=<type>&filename=<fname>&coll=<coll>&disp=<disp>&div=<div>&inst=<inst>]]></url>
    <url type="write"><![CDATA[http://localhost/cgi-bin/fileupload.php]]></url>
    <url type="delete"><![CDATA[http://localhost/cgi-bin/filedelete.php?&filename=<fname>&coll=<coll>&disp=<disp>&div=<div>&inst=<inst>]]></url>
</urls>

The symbols that are substituted with values are:

Symbol Description
<fname> The intended name of the file on the server side.
<coll> The name of the Collection the attachment belongs.
<disp> The name of the Discipline the attachment belongs.
<div> The name of the Division the attachment belongs.
<inst> The name of the Institution the attachment belongs.
<type> The type of attachment 'O' for original file. 'T' for thumbnail.
scale There is no substitutable token '<scale>'. The calling implementation can optionally add the argument 'scale=XXX&' where XXX is the size in pixels of the largest side and the scaling MUST maintain the aspect ratio.

The 'read' and 'delete' URLs can be RESTful or HTTP Get, the substitution function does not make any assumptions about how they are constructed. The 'write' URL is constructed as a HTTP Post containing the file and the names/value pairs as parts.

Local Cache

As Specify interacts with the AWS it uses a local short to term cache to store the attachments. This enables users to double-click on the attachment to launch a local viewer application. The cache is also used as temporary storage for thumbnails and when files are being uploaded.

The Back-end

The back-end scripts can be any scripting language like PHP or a Java Servlet.

 

Specify Thick-Client Web Attachment Module API

NOTE: The initial requirement for the Attachment Manager was for a non-techical person to be able to install Specify and then configure a simple perference for where they wanted the attachments to be stored, either on the local disk or on a shared network drive. The initial implementation pre-dated 'cloud storage' and it was unreasonable to require any type of server software to be installed. These requirements also meant that all attachment imformation would be stored and maintained on the 'client side' in the Specify database by Specify.

The following API was developed in the early days of Specify before a complete understanding of how it needed to interact with the thick-client and any future thin-client applications. A file-based implementation was developed first and there were not any web-based implementations until 2012. There is still a requirement for both a 'local' file-based and a web-based implementation.

Problems with the current API:

 

public interface AttachmentManagerIface
{
    /**
     * Whether it was initialized OK.
     * @param urlStr null for File-based stores and not null for web-based stores
     * @return true if initialized and can be used.
     */
    public abstract boolean isInitialized(String urlStr);
    
    /**
     * Sets the attachmentLocation field in the passed in Attachment
     * object.  This allows the AttachmentManagerIface implementation
     * to provide a storage location that isn't already in use.
     * 
     * @param attachment the Attachment for which a storage location is needed
     * @param doDisplayErrors false for silent mode, true for popu dialog errors
     * @return true if successfully set
     */
    public abstract boolean setStorageLocationIntoAttachment(Attachment attachment, boolean doDisplayErrors);
    
    /**
     * Get a file handle to the attachment original.
     * 
     * @param attachment the attachment record
     * @return a java.io.File handle to the attachment document
     */
    public abstract File getOriginal(Attachment attachment);
    
    /**
     * @param attachLoc
     * @param originalLoc
     * @param mimeType
     * @return
     */
    public abstract File getOriginal(String attachLoc,
                                     String originalLoc,
                                     String mimeType);

    /**
     * @param attachLoc
     * @param originalLoc
     * @param mimeType
     * @param maxSideInPixels
     * @return
     */
    public abstract File getOriginalScaled(String attachLoc,
                                           String originalLoc,
                                           String mimeType,
                                           int maxSideInPixels);

    /**
     * @param attachmentID the record id of the attachment.
     * @return the embedded image meta data for the image in the repository.
     */
    public abstract String getMetaDataAsJSON(int attachmentID);
    
    /**
     * @param attachmentID the record id of the attachment.
     * @return the embedded image meta data for the image in the repository.
     */
    public abstract Calendar getFileEmbddedDate(int attachmentID);
    
    /**
     * Get a file handle to the attachment thumbnail.
     * 
     * @param attachment the attachment record
     * @return a java.io.File handle to the attachment thumbnail
     */
    public abstract File getThumbnail(Attachment attachment);
    
    
    /**
     * Get a file handle to the attachment thumbnail.
     * 
     * @param attachmentLoc contents of the 'AttachmentLocation' column in the database.
     * @param mimeType the mimeType of the file.
     * @return File handle if there is one
     */
    public abstract File getThumbnail(String attachmentLoc, String mimeType);
    
    /**
     * Store a new attachment file (and thumbnail) in the manager's storage area.  A call
     * to attachment.setAttachmentLocation will occur.
     * 
     * @param attachment the attachment record
     * @param attachmentFile the original attachment document
     * @param thumbnail the thumbnail of the original
     * @throws IOException if an error occurs when storing the files
     */
    public abstract void storeAttachmentFile(Attachment attachment, File attachmentFile, File thumbnail) throws IOException;
    
    /**
     * Replace the existing attachment file with a new version.  If an exception occurs during the
     * replacement process, there is no guarantee as to the state of the attachment file or its
     * thumbnail.
     * 
     * @param attachment the attachment record
     * @param newOriginal the new version of the attachment document
     * @param newThumbnail the new version of the thumbnail
     * @throws IOException if an error occurs when replacing the files
     */
    public abstract void replaceOriginal(Attachment attachment, File newOriginal, File newThumbnail) throws IOException;
    
    /**
     * Delete the files (original and thumbnail) associated with this attachment record.
     * 
     * @param attachment the DB record holding the file info
     * @throws IOException if an error occurs when deleting the files
     */
    public abstract void deleteAttachmentFiles(Attachment attachment) throws IOException;
    
    
    /**
     * Regenerates the thumbnail from the original file.
     * @param attachment the attachment with thumbnail
     * @return the File for the thumbnail.
     * @throws IOException
     */
    public abstract File regenerateThumbnail(final Attachment attachment) throws IOException;
    
    /**
     * Resets the baseDirectory.
     * @param baseDir the new base directory
     * @throws IOException
     */
    public abstract void setDirectory(File baseDir) throws IOException;
    
    /**
     * @return the directory of the Attachment Manager
     */
    public abstract File getDirectory();
    
    /**
     * @return true if a networked mapped drive is being used for the attachment manager. false if a Web Service is being used.
     */
    public abstract boolean isDiskBased();

    /**
     * @return string URI 'template' that includes symbols for substituting various parameters that can be substituted with the 
     * appropriate values for locating the proper image. Any service or app can ask for this URL template, the Attachment Manager does not
     * have to support all of the symbols listed below, but they may be passed in.
     * FILENAME - name of the file.
     * COLL - Collection Name  (same as in Specify)
     * DIV - Division Name (same as in Specify)
     * INST - Institution Name (same as in Specify)
     * SCALE - an integer number in pixels indicating the size of the largest side of the image. Blank for no scaling.
     */
    public abstract String getImageAttachmentURL();
    
    /**
     * @param sizeInPixels
     */
    public void setThumbSize(int sizeInPixels);
    
    /**
     * Perform any internal cleanup needed before shutdown.
     */
    public abstract void cleanup();
}


 

Java Thumbnailer API

This interface / API enables multiple 'thumbnail generators' to be installed per mime-type. The idea here is that Java implementations would be used to implement thumbnail generation for a given mime-type and if it didn't exist or wasn't possible, it would return with a standard icon for that mime-type. Once again, since there would not be a web-based server all thumbnails would be generated locally. THe next Specify release 6.5 will include a PDF thumbnailer that will generate an image form the first page of the PDF.

We need to think through how and where the thumbnails will be generated in the future for the various different mime-types.

public interface ThumbnailGeneratorIFace
{
	/**
	 * Sets the maximum width of any visual thumbnails created.
	 *
	 * @param maxWidth the maximum thumbnail width
	 */
	public void setMaxWidth(int maxWidth);

	/**
	 * Sets the maximum height of any visual thumbnails created.
	 *
	 * @param maxHeight the maximum thumbnail height
	 */
	public void setMaxHeight(int maxHeight);
	
	/**
	 * Sets the maximum duration of any audio or video 'thumbnails' created.
	 *
	 * @param seconds the time length of the audio or video thumbnails created
	 */
	public void setMaxDuration(int seconds);

	/**
	 * Sets the quality factor for any thumbnailers that implement a configurable
	 * level of quality.
	 *
	 * @param percent the quality factor
	 */
	public void setQuality(float percent);

	/**
	 * Returns an array of MIME types that are supported by this thumbnail generator.
	 *
	 * @return the array of supported MIME types
	 */
	public String[] getSupportedMimeTypes();

	/**
	 * Create a thumbnail for the given original, placing the output in the given output file.
	 *
	 * @param originalFile the path to the original
     * @param thumbnailFile the path to the output thumbnail
     * @param doHighQuality true creates a high quality thumbnail (slow), false a low resolution thumbnail (fast)
     * @return true if thumbnail was create and false it is wasn't
	 * @throws IOException if any IO errors occur during generation or storing the output
	 */
	public boolean generateThumbnail(String originalFile, 
	                                 String thumbnailFile,
	                                 boolean doHighQuality) throws IOException;
}


JSON Results From the Attachment Manager

In the AttachmentManagerIface interface the method 'getMetaDataAsJSON' returns a JSON encoded string that represents the internal image meta (e.g. EXIF, etc.). The JSON is what was retruned from our PHP script using the 'exif_read_data' call. The current Java implementation uses the 'metadata-extractor.jar.'

[
  {"Name" : "FILE",
   "Fields" : {
    "FileName" : "sp62405537498279011105.att.JPG",
    "FileDateTime" : "1351181117",
    "FileSize" : "1271528",
    "FileType" : "2",
    "MimeType" : "image/jpeg",
    "SectionsFound" : "ANY_TAG, IFD0, THUMBNAIL, EXIF, INTEROP, MAKERNOTE"
  }},
  {"Name" : "COMPUTED",
   "Fields" : {
    "html" : "width=3456 height=2304",
    "Height" : "2304",
    "Width" : "3456",
    "IsColor" : "1",
    "ByteOrderMotorola" : "1",
    "CCDWidth" : "22mm",
    "ApertureFNumber" : "f/6.3",
    "UserComment" : "",
    "UserCommentEncoding" : "UNDEFINED",
    "Thumbnail.FileType" : "2",
    "Thumbnail.MimeType" : "image/jpeg"
  }},
  {"Name" : "IFD0",
   "Fields" : {
    "Make" : "Canon",
    "Model" : "Canon EOS DIGITAL REBEL XT",
    "Orientation" : "1",
    "XResolution" : "72/1",
    "YResolution" : "72/1",
    "ResolutionUnit" : "2",
    "Software" : "Microsoft Windows Photo Viewer 6.1.7600.16385",
    "DateTime" : "2012:03:19 11:52:12",
    "YCbCrPositioning" : "2",
    "Exif_IFD_Pointer" : "2322",
    "UndefinedTag:0xEA1C" : ""
  }},
  {"Name" : "THUMBNAIL",
   "Fields" : {
    "Compression" : "6",
    "XResolution" : "72/1",
    "YResolution" : "72/1",
    "ResolutionUnit" : "2",
    "JPEGInterchangeFormat" : "13584",
    "JPEGInterchangeFormatLength" : "3676"
  }},
  {"Name" : "EXIF",
   "Fields" : {
    "ExposureTime" : "1/160",
    "FNumber" : "63/10",
    "ExposureProgram" : "0",
    "ISOSpeedRatings" : "400",
    "ExifVersion" : "0221",
    "DateTimeOriginal" : "2012:03:15 06:20:16",
    "DateTimeDigitized" : "2012:03:15 06:20:16",
    "ComponentsConfiguration" : "",
    "ShutterSpeedValue" : "479850/65536",
    "ApertureValue" : "348042/65536",
    "ExposureBiasValue" : "0/2",
    "MeteringMode" : "5",
    "Flash" : "16",
    "FocalLength" : "26/1",
    "MakerNote" : "",
    "UserComment" : "",
    "FlashPixVersion" : "0100",
    "ColorSpace" : "1",
    "ExifImageWidth" : "3456",
    "ExifImageLength" : "2304",
    "InteroperabilityOffset" : "13440",
    "FocalPlaneXResolution" : "3456000/874",
    "FocalPlaneYResolution" : "2304000/582",
    "FocalPlaneResolutionUnit" : "2",
    "CustomRendered" : "0",
    "ExposureMode" : "0",
    "WhiteBalance" : "0",
    "SceneCaptureType" : "0",
    "UndefinedTag:0xEA1C" : "",
    "UndefinedTag:0xEA1D" : "4210"
  }},
  {"Name" : "INTEROP",
   "Fields" : {
    "InterOperabilityIndex" : "R98",
    "InterOperabilityVersion" : "0100"
  }},
  {"Name" : "MAKERNOTE",
   "Fields" : {
    "ModeArray" : "Array",
    "UndefinedTag:0x0002" : "Array",
    "UndefinedTag:0x0003" : "Array",
    "ImageInfo" : "Array",
    "ImageType" : "",
    "FirmwareVersion" : "",
    "OwnerName" : "",
    "Camera" : "1320741137",
    "UndefinedTag:0x000D" : "",
    "CustomFunctions" : "Array",
    "UndefinedTag:0x0010" : "-2147483255",
    "UndefinedTag:0x0012" : "Array",
    "UndefinedTag:0x0013" : "Array",
    "UndefinedTag:0x0015" : "-1610612736",
    "UndefinedTag:0x0019" : "1",
    "UndefinedTag:0x0083" : "0",
    "UndefinedTag:0x0093" : "Array",
    "UndefinedTag:0x00A0" : "Array",
    "UndefinedTag:0x00AA" : "Array",
    "UndefinedTag:0x00D0" : "0",
    "UndefinedTag:0x00E0" : "Array",
    "UndefinedTag:0x4001" : "Array",
    "UndefinedTag:0x4002" : "Array",
    "UndefinedTag:0x4003" : "Array"
  }}
]