Tutorial: Common Analysis Mistakes

Digital photo forensics is a complex task. There is no one-button solution for highlighting modifications or determining if a picture is real or fake. This task is made more difficult by some very common mistakes.

1. Mistake #1: Asking the Wrong Question
2. Mistake #2: Forcing an Answer
3. Mistake #3: Ignoring Size and Quality
4. Mistake #4: Modifying Content (Resaves, Adjustments, and Annotations)
5. Mistake #5: Capturing Screenshots

Mistake #1: Asking the Wrong Question

What may seem like an easy question may actually be very complicated. Is this "real"? Was this modified? Did this really come from a ShutterSnap D-9050 digital camera? Or is that really her head on that body?

There are two important things to remember when evaluating pictures:

1. The analysis algorithms and tools at FotoForensics evaluate the picture, but not the content. For example, if you print out a fake document and then photograph it, then you have a real photo of a fake document -- it is still a real photo.
   
   These tools may be able to help you identify any digital alterations to a picture, but not any special circumstances prior to capturing the photo. (FotoForensics will not tell you if that UFO is actually a hub cap thrown into the air, but it will help you determine if the picture was digitally altered.)
2. The algorithms and analysis methods extract information from files. You may want one question answered, but the tools may address a different question. For example, Error Level Analysis (ELA) quantifies the JPEG compression rate across the image. The compression rate for similar surfaces should look similar, and the rates for similar edges should look similar. ELA answers the question: "What is the JPEG error level potential (the compression rate) across the picture?"
   
   Any inconsistency, such as similar high contrast edges with very different ELA results, denotes that something was changed. But ELA does not identify the cause -- it only highlights artifacts and areas that an analyst can identify as inconsistent.

The analyst takes all of the clues identified by the various analysis methods and tries to find a consistent explanation. This is the basis of the scientific method:

1. Observe. We identify elements and artifacts in the image file.
2. Question. We ask questions related to the observations. For example, "is this digitally altered?", "was she really there?", or "where did this file come from?"
3. Hypothesize. We construct a testable hypothesis that addresses the questions.
4. Predict. Given the hypothesis, we predict the expected outcome.
5. Test. We test the hypothesis with one or more reproducible experiments.
6. Evaluate. We compare the experiment's results against the expected outcome. If they match, then the hypothesis is plausible. (Plausible means it is an option, but it is not a confirmation.) If they do not match, then we can rule out the hypothesis; a false hypothesis can be confirmed. The results may lead to more questions that repeat this cycle.

In many cases, it is easier to test the opposite of what you want by using a null hypothesis. For example, you may want to know if the picture was modified, but it may be easier to test whether the picture is camera-original. Any detectable modifications will identify a false hypothesis. If you can show that it is not original, then you can confirm that it was resaved and potentially modified. A negative result can then be refined to determine if the picture was resaved, stripped, or intentionally altered.

By the same means, a forensic crime scene examiner may look at blood spatter on a wall and deduce homicide or suicide, physical attributes of the attacker, and even the order of events. However, blood spatter tests do not directly identify any of these conclusions. Instead, the tests identify basic clues. For example, the shape of the drops can identify the direction of travel. The size and quantity of droplets identify the type of blood source (e.g., a deep cut or arterial spray), and a lack of droplets may indicate something that prevented the blood from hitting the wall (like an attacker's body in the way). The examiner collects all of these findings and deduces a plausible scenario.

For digital photo forensics, analysts must be careful to ask specific questions that can be tested and compared against a potential scenario. An investigator must also remember to ask: is this the only explanation?

Mistake #2: Forcing an Answer

It is very common to see an amateur analyst force a result into their desired answer. For example, ELA renders a picture that reflects the compression rate. People often post to Twitter comments like "It's real!" or "It's fake!" when the ELA result really indicates a very low quality image or a consistent compression rate. (With ELA, "white" does not mean modified; white means a higher error level potential that must be compared against similar edges and similar surfaces in the picture.)

For example, a photo may show a white edge around a person's hair that stops at the body. This could mean that the head was spliced onto the body. However, it could also identify selective sharpening, editing of the head, editing around the head, or a high contrast between the hair and the background. The picture may also be scaled or resaved by an Adobe application -- both of which could increase the error potential along high-contrast and high-frequency edges. ELA shows where the compression level varies within the picture, but it does not identify what caused the variation.

Similar forced answers are commonly seen with metadata. For example, the metadata may identify that Adobe Photoshop was used. People commonly jump to the conclusion that "Photoshop" means altered in a misleading way (maliciously altered). However, that is not always true. If a user wants to prepare a picture for the web, then they are just as likely to use Photoshop as any other program. The presence of Photoshop in the metadata only identifies the tool that was used and not any malicious modifications to the picture's content.

Except in extreme cases (like when ELA strongly identifies one area as being significantly different), declaring a conclusion based on one test result usually indicates someone forcing an answer.

Mistake #3: Ignoring Size and Quality

The very first question an analyst should ask is "where did this picture come from?" Online services, like Facebook and Twitter, resave pictures at a low quality. In addition, Facebook and Twitter do not generate pictures; they only distribute pictures that came from somewhere else. A picture that has been passed around is likely to be repeatedly resaved, resized, and otherwise altered.

The size of an image and the quality of the picture directly impacts the ability to evaluate the file. While a large picture that is near-camera-original may reveal a wealth of information, a tiny thumbnail image is unlikely to tell much about the picture. A large picture that has been repeatedly resaved with JPEG compression is also unlikely to have subtle artifacts intact.

As an analogy to pulling clues out of images, consider tracking someone's footprints on the ground. If the soil is soft and retains shape (like a recently plowed field), then you can probably see every detail about each footstep and even identify the shoe's tread. A JPEG resave is like a light rain -- it obscures some of the details. Multiple resaves are like a heavy rain -- you may see the footsteps but none of the details. But evaluating a picture that is tiny and low quality? That's like tracking footsteps along a sandy beach during a hurricane -- you probably will not be able to identify any footprints.

Extracting fine details from very tiny icons, avatars, and thumbnail images is like reading tea leaves. If you are right, it's probably due more to coincidence than skill.

This does not mean that you cannot evaluate pictures from Facebook or Twitter. However, you need to remember the source. A low quality picture or a small image may mean that you cannot conclusively answer questions regarding modifications. The more extreme the modification, the more likely it is to be detected in a low quality picture.

Mistake #4: Modifying Content (Resaves, Adjustments, and Annotations)

The last thing an investigator wants to do is modify the evidence. Every modification, every save, and every annotation results in a change to the data. Even if you do not intentionally edit the picture, anything other than a byte-per-byte copy results in a modification to the file.

One of the most common mistakes happens when people pass evidence to an investigator. They may scale the picture larger, brighten the image, or annotate it with circles and arrows so that the investigator knows where they should be looking. Pictures may also be spliced together (side-by-side) or given an attractive border.

However, each of these alterations fail to retain the integrity of the evidence. The user may think that they are helping by making something easier to see, but they are really altering the evidence: obscuring potentially critical details in the image, lowering the quality with a resave, and stripping metadata.

Annotations, highlighting, and other alterations do not help investigators. This is one reason why police officers cordon off crime scenes. If the public is permitted to continually walk through an active crime scene, then they are likely to disturb evidence. Drawing arrows and circles into a picture to highlight elements is an alteration; analysis will easily identify the annotations, while the annotations and resaves are likely to wipe away trace evidence related to the source picture. If the picture does need some kind of enhancement, then the investigator will do it in a way that does not alter the source file.

Alterations are also common for pictures found online. An original photo may be resized for the web (modification #1), uploaded to Facebook (modification #2), downloaded, cropped (#3), uploaded to Imgur (#4), copied from Imgur, brightened (#5), and posted to Twitter (#6), and so on. A viral photo can quickly undergo dozens or hundreds of alterations. Each modification changes the image and makes evaluating the content more difficult.

For an investigator, it is best to get any picture directly from the source. A picture that has been passed around on Facebook and Twitter is unlikely to have many fine details left.

In some cases, modifications to evidence may be unintentional. A user who passes along a file may not know how to transfer it without using a program like Photoshop or Microsoft Photo Viewer. They might not realize that Imgur strips metadata or that Facebook resaves all images at a low quality. If they don't know, then they will not realize that they have modified the picture.

Similarly, splicing pictures for a side-by-side example or annotating images with copyright statements, URLs, or red circles may seem like a good idea to the user. If the user strongly believes that they must annotate the image, then they should copy the file to a different name and only annotate the copy. They should send the investigator both the annotated and unmodified source files.

Mistake #5: Capturing Screenshots

Saving pictures from web sites or extracting images from files may not always be straightforward. Some web sites use complex JavaScript or HTML tricks to deter people from extracting pictures. And images in a PDF or other file formats may require special tools for extraction.

Smartphones are notoriously bad at saving pictures for analysis. Many users do not know how to browse their smartphone's file system. And even if they can view the file system, they may not know where the pictures are saved. Even attaching a USB cable for file transfers may be overly complicated -- it all depends on the smartphone and the user's technical abilities.

What users typically end up doing is taking a screenshot. While screenshots capture what was on the screen, the application captured by the screenshot likely altered the picture as it was displayed on the screen. Applications like web browsers and PDF viewers typically scale the page to fit on the screen. However, even viewing the page at "100% size" may still result in the scaling of embedded pictures. For example, a web page may scale a picture for display, and then the web browser may further scale the page to fit the screen.

When a picture is scaled, every pixel is modified based on the scaling factor. This modification impacts the entire image. While this does not change the image stored within the source file, a screen capture ignores the source file. The screenshot of a web page will capture the scaled and altered image displayed by the browser.

Screenshots strip out metadata. This removes an entire analysis dimension. Screen captures may also introduce resave artifacts if the screenshot is ever saved as a JPEG. (And don't trust TIFF since some TIFF files use JPEG encoding.) The application captured by the screenshot may further alter the picture by applying color profiles or gamma corrections. Anything that alters the colors or size of a picture is a modification to the image. And modifications obscure details. In effect, an analysis of a screen capture is likely to detect the screen capture software, information about the screen, and any display artifacts introduced by the application that displayed the image; an analysis is unlikely to detect any alterations hidden within the source picture.

When it comes to analyzing pictures from a screenshot, don't do it. And if you have no alternative, then be aware that the evaluation is more likely to identify artifacts from the screen capture software than anything found in the picture.

Variations of the screen capture mistake include:

• Print-and-scan mistake. When a photo is printed, it becomes a very low quality image. Scanning in the picture introduces scanner artifacts. The net result is that an analyst will likely identify information about the scanner and the printer, and nothing about the image's content.
• Video frames mistake. The highest quality video is typically lower quality than a low quality JPEG. Video players also scale the picture and alter colors in images -- these are significant post-processing steps. Extracting frames from video for analysis will result in no original metadata and a low quality picture that has been significantly post-processed. It is unlikely to provide useful information about the video's content.

Mitigating Mistakes

While these five types of mistakes are very common, analysts do have options to mitigate some of these problems. For example:

Mitigation	Explanation
1. Knowing	By understanding these issues, an analyst can better identify these situations.
2. Accepting uncertainty	When evaluating an image, the result may not be a "yes" or "no" answer. Responses like "inconclusive", "cannot be determined because..." or "I cannot tell due to..." are perfectly acceptable.
3. Offering alternate answers	Every picture tells you something. Even a very low quality picture can be informative. For example, you can ask "why is it such as low quality picture?" More often than not, an investigator can point out inconsistencies in the assumptions that drive the questions. If the image is supposed to be direct from an authoritative source, then it should not be very low quality.
4. Finding better sources	A small or cropped version of a photo had to come from somewhere. Where did the picture come from? Similar image search tools, such as TinEye and Google Image Search, may be able to find larger versions of the picture, versions with more content (uncropped), or at a higher quality. (See the Similar Image Search tutorial for more information about finding visually similar pictures.)
5. Requesting original sources	Screenshots record what was displayed on the screen, which is not ideal for an evaluation. However, the screenshot may contain enough information for you to track down a higher quality source. For example, if the screenshot shows a web page, then go directly to the web page for the content to evaluate.

A forensic evaluation should be consistent and repeatable. However, the strength of the conclusion depends on the quality of the data being analyzed. Is the picture from an authoritative source? Is it the highest quality available? Are the test results consistent with the conclusions? Or did the evaluation include one of these five common mistakes?